Newsgroup discussion of "observed" power from SPSS, which is the basis of the D'Amico et al. method of sample size estimation. ------------------------------------------------------------------------------------- From: Bruce Weaver Newsgroups: sci.stat.edu Subject: Re: help: sample required for 2x2 factorial design analysis of variance Date: Mon, 15 Nov 2004 15:31:12 -0500 Message-ID: <2vsi0bF2ostn9U1@uni-berlin.de> References: <8b3f83cd.0411150831.2512f89f@posting.google.com> Richard Ulrich wrote: -- snip -- > > - A tool like the SPSS estimator of power can be very > misleading about power when the d.f. in the example is > small -- Unless that has been changed in recent years. > I posted about this a few years ago in a discussion in > the comp.stat-sys.spss group, with examples. > Is this the post you're referring to, Rich (mind the word-wrap)? http://groups.google.ca/groups?q=Richard+Ulrich+power+SPSS&hl=en&lr=&selm=5imb4d%24qhe%40usenet.srv.cis.pitt.edu&rnum=1 http://groups.google.ca/groups?selm=5imb4d%24qhe%40usenet.srv.cis.pitt.edu&output=gplain -- Bruce Weaver bwea...@lakeheadu.ca www.angelfire.com/wv/bwhomedir ------------------------------------------------------------------------------------- NNTP-Posting-Date: Tue, 16 Nov 2004 13:38:11 -0600 From: Richard Ulrich Newsgroups: sci.stat.edu Subject: Re: help: sample required for 2x2 factorial design analysis of variance Date: Tue, 16 Nov 2004 14:38:11 -0500 Yes, that's one. The shorter reference gets article by itself (How do you do that?) and the longer one links to the original thread. The SPSS code was generating random data as if it were 'pilot data', and then showing that the SPSS procedure reports very high power for a very-non-significant result. This seeming-paradox is a consequence of how the 'effect sizes' do not take chance into account. You will *observe* a large R-squared with many variables and few cases; an *underlying* R-squared that is large will have pretty good power -- so that is what SPSS reported. David Nichols said then, other programs do the same, which is a reason to be careful about using any of them. That was 1997. I haven't looked at what SPSS does now. Nobody mentioned any changes in the basis, that I recall. -- Rich Ulrich, wpi...@pitt.edu http://www.pitt.edu/~wpilib/index.html ------------------------------------------------------------------------------------- From: neilane...@msn.com (Neila Nessa) Newsgroups: sci.stat.edu,comp.soft-sys.stat.spss Subject: Re: SPSS power calculation with MANOVA and MACRO (Break out the hatchet ;-) Date: 17 Nov 2004 18:36:01 -0800 Bruce Weaver wrote in message news:<419A63CD.8080906@lakeheadu.ca>... > [ cross-posted to comp.soft-sys.stat.spss ] > -Inline- > > Interesting. Looks like we should be having this (re-)discussion in > comp.soft-sys.stat.spss. Maybe we can get some commentary from the > current SPSS tech support folks. Are you there, Jon Fry? Jon is not a member of SPSS Technical Support. He is one of the lead SPSS developers and is basically omniscient with respect to all the greasy grimy stuff the backend is up to. He probably knows more than anyone about what goes on under the hood of the beast. He jumps in to this ng from time to time to help all us deaf, dumb and blind kids play pinball ;-), but he's definitely not tech support. (Unless he's gone into some warped Bodhisattva modality ;-))) LOL! Hope you are well Jon in case you are lurking - say Hi to Tex-! David Marso/Neila ------------------------------------------------------------------- A few comments on Rich's-Bruce's MACRO (and a refined version ;-). I was about to get the heck out of here but I got sucked in big time (Damn it, thanks again Bruce, your fault ;-) !!! I figured hmmm, add N and P as parameters to the macro (easy). Hmm, but I don't want the last variable in the list, Simply use P-1 cept macro is a numeritard. Teach it math? Then why not also generalize the names, hmmm maybe we can nix MANOVA and use GLM (OK, we get a match) , Hmmm why run another procedure to get means when we can get those cute little pivot tables from GLM. Oops they are not compact the way we would like: Hmmm Autoscript, Darn, the output isn't properly refreshed Break out the hatchet and hack our way through the freaking objects again! So, an hour and a half later this is finally getting a proper send off!!! This one is neat because it demonstrates alot of the capability of the MACRO facility (DEFAULT parameters,Conditional execution, hacking out a NON-FEATURE subtraction (WAKE UP SPSS!!!), Iteration, Complex CONCATENATION etc..... Homework assignment for all you warped macro lovers out there ;-) Make this bad dog roll over and do addition, multiplication and division hmmm, how about a square root, PI to 10 digits....... Have fun ;-))) David/Neila -SchizoSPSSer- Long story -No I don't care to elaborate- ----------------------------------------------------------------------- /***************************************************************/ /* Macro for simulation of Power calculations and comparison */. /* of MANOVA and GLM */. /***************************************************************/. /* User can vary the Number of cases (N), */. /* Number of Variables (P), */. /* Grouping variable name (GRP) */. /* Number of Groups (NGPS), */. /* Rootname for new variables (ROOT), */. /* Alpha (ALPHA), */. /* Random SEED for simulation (SEED) */. /***************************************************************/. /* David Marso's rewrite of Bruce Weaver's rewrite of Rich */. /* Ulrich's Macro from ??? */. /***************************************************************/. /* Substituted GLM for MANOVA and nixed of MEANS Procedure */. /* Replaced RANK function to use NTILES rather than PERCENT */. /* Followed by RECODE. */. /***************************************************************/. /* Usage SIMPWR N 50 P 6 NGPS = 10 ALPHA=.01 */. /* SIMPWR */(Uses default parameter values) */. /***************************************************************/. DEFINE SIMPWR (N !TOKENS(1) !DEFAULT (50) / P !TOKENS(1) !DEFAULT (6) / GRP !TOKENS(1) !DEFAULT (GRP) / NGPS !TOKENS(1) !DEFAULT (10) / ROOT !TOKENS(1) !DEFAULT (VAR) / ALPHA !TOKENS(1) !DEFAULT (.05) / SEED !TOKENS(1) !DEFAULT (7000111) ) TITLE random data, N= !N , P = !P . set seed=!SEED . * Simulate a bunch of Normals for N cases P variables *. new file. input program. loop id=1 to !N. + do repeat VAR=!CONCAT(!ROOT,1) TO !CONCAT(!ROOT,!P). compute VAR=normal(10). FORMATS VAR (F5.2). end repeat. end case. end loop. end file. end input program. formats id(f5) . * Create a grouping variable * . RANK !CONCAT(!ROOT,!P) /ntiles(!NGPS) INTO !GRP. ** Let's teach that numerically challenged string parser to compute P-1 ** . ** Break out the hatchet, hack hack hack!!! !LET !SIZE=!NULL !DO !I=1 !TO !P !IF (!I !LT !P) !THEN !LET !SIZE=!CONCAT(!SIZE,"X") !IFEND !DOEND !LET !VARSPEC=!CONCAT(!ROOT,1,' TO ',!ROOT,!LENGTH(!SIZE), ' by ',!grp) GLM !VARSPEC /PRINT = DESCRIPTIVE OPOWER /CRITERIA = ALPHA(!ALPHA) /DESIGN = !grp . *MANOVA !VARSPEC (1,!NGPS) * /print=signif(hypoth) CELLINFO(MEANS) * /power=F(.05) . *MEANS !VARSPEC/cells = mean. !enddefine. SET PRINTBACK ON MPRINT ON . SIMPWR N 50 P 6 NGPS = 10 ALPHA=.05 . Same as writing (since we provide defaults in macro definition). SIMPWR . If you get rid of the MANOVA and MEANS command you can still get the MEANS like format from GLM by using the following autoscript. ---- Sub GLM_Table_DescriptiveStatistics_Create(objTable As Object, objOutputDoc As Object, lngIndex As Long) 'Autoscript 'Trigger Event: DescriptiveStatistics Table Creation after running GLM procedure. Dim objPivot As PivotTable Dim objOutput As ISpssOutputDoc Dim objItem As ISpssItem Dim N As Long Dim I As Long Set objPivot =objTable With objPivot With .PivotManager .ColumnDimension(0).MoveToLayer(0) .RowDimension(0).MoveToColumn(0) End With End With 'THIS IS A HACK BECAUSE SPSS DOES NOT PROPERLY REFRESH THE SCREEN!!! Set objOutput = objOutputDoc With objOutput.Items N=.Count For I=N-1 To 1 Step -1 Set objItem =.GetItem(I) If objItem.Label ="Descriptive Statistics" Then objItem.Activate objItem.Deactivate End If Next End With End Sub -------------------------------------------------------------------------------------