Ext2fs Undeletion mini-HOWTO **************************** by Aaron Crane v1.0, Sat Jan 18 1997 Contents ======== 0. Introduction 1. How *not* to delete files 2. What recovery rate can I expect? 3. So, how do I undelete a file? 4. Unmounting the filesystem 5. Preparing to change inodes directly 6. Preparing to write data elsewhere 7. Finding the deleted inodes 8. Obtaining details of the inodes 9. Recovering data blocks 10. Modifying inodes directly 11. Will this get easier in future? 12. Are there any tools to automate this process? Appendices ---------- A. Colophon B. Credits C. Bibliography D. Legalities 0. Introduction ================ This mini-Howto attempts to provide hints on how to retrieve deleted files from an ext2 filesystem. There is also a limited amount of discussion of how to avoid deleting files in the first place. I intend it to be useful certainly for people who have just had, shall we say, a little accident with `rm'; however, I also hope that people read it anyway. You never know -- one day, some of the information in here could save your bacon. The text assumes a little background knowledge about Unix filesystems in general; however, I hope that it will be accessible to most Linux users. If you are an outright beginner, I'm afraid that undeleting files under Linux *does* require a certain amount of technical knowledge and persistence, at least for the time being. You will be unable to recover deleted files from an ext2 filesystem without at least read access to the raw device on which the file was stored. In general, this means that you must be root. You also need `debugfs' from the `e2fsprogs' package. This should have been installed by your distribution. Why have I written this? It stems largely from my own experiences with a particularly foolish and disastrous `rm -r' command as root. I deleted about 97 JPEG files which I needed and could almost certainly not recover from other sources. Using some helpful tips (see Appendix B, `Credits') and a great deal of persistence, I recovered 91 files undamaged. I managed to retrieve at least parts of five of the rest (enough to see what the picture was in each case). Only one was undisplayable, and even for this one, I am fairly sure that no more than 1024 bytes were lost (though unfortunately from the beginning of the file; given that I know nothing about the JFIF file format I had done as much as I could). I shall discuss further below what sort of recovery rate you can expect for deleted files. 1. How *not* to delete files ============================= It is vital to remember that Linux is not MS-DOG. For MS-DOG (including Windoze 95), it is generally fairly straightforward to undelete a file -- the `operating system' (I use the term loosely) even comes with a utility which automates much of the process. For Linux, this is not the case. So. Rule number one (the prime directive, if you will) is: *** KEEP BACKUPS *** no matter what. I know, I'm a fine one to talk. I shall merely plead impoverishment (being a student must have *some* perks) and exhort all right-thinking Linux users to go out and buy a useful backup device, work out a decent backup schedule, and to *stick to it*. For more information on this, see Frisch (1995). In the absence of backups, what then? (Or even in the presence of backups: belt and braces is no bad policy where important data is concerned.) Try to set the permissions for important files to 440 (or less): denying yourself write access to them means that `rm' requires an explicit confirmation before deleting. (I find, however, that if I'm recursively deleting a directory with `rm -r', I'll interrupt the program on the first or second confirmation request and reissue the command as `rm -rf'.) A good trick for selected files is to create a hard link to them in a hidden directory. I heard a story once about a sysadmin who repeatedly deleted /etc/passwd by accident (thereby half-destroying the system). One of the fixes for this was to do something like the following (as root): # mkdir /.backup # ln /etc/passwd /.backup It requires quite some effort to delete the file contents completely: if you say # rm /etc/passwd then # ln /.backup/passwd /etc will retrieve it. Of course, this does not help in the event that you overwrite the file, so keep backups anyway. Some people advocate making `rm' a shell alias or function for `rm -i' (which asks for confirmation on *every* file you delete). Personally, I cannot stand software which won't run unattended, so I don't do that. There is also the problem that sooner or later, you'll be running in single-user mode, or using a different shell, or even a different machine, where your `rm' function doesn't exist. If you expect to be asked for confirmation, it is easy to forget where you are and to specify too many files for deletion. Likewise, the various scripts and programs that replace `rm' are, IMHO, very dangerous. A slightly better solution is to start using a package which handles `recyclable' deletion without using any variant of the `rm' command. For details on these, see Peek, et al (1993). 2. What recovery rate can I expect? ==================================== That depends. Among the problems with recovering files on a high-quality, multi-tasking, multi-user operating system like Linux is that you never know when someone wants to write to the disk. So when the operating system is told to delete a file, it assumes that the blocks used by that file are fair game when it wants to allocate space for a new file. (This is a specific example of a general principle for Linux: the tools and kernel assume that the users aren't idiots.) In general, the more usage your machine gets, the less likely you are to be able to recover files successfully. Also, disk fragmentation can affect the ease of recovering files. If the partition containing the deleted files is very fragmented, you are unlikely to be able to read a whole file. If your machine, like mine, is effectively a single-user workstation (mine doesn't even have a net connection yet; maybe next year), and you weren't doing anything disk-intensive at the fatal moment of deleting those files, I would expect a recovery rate in the same ball-park as detailed above. I retrieved nearly 94% of the files (and these were binary files, please note) undamaged. If you get 80% or better, you can feel pretty pleased with yourself, I should think. 3. So, how do I undelete a file? ================================= The procedure principally involves finding the data on the raw partition device and making it visible again to the operating system. There are basically two ways of doing this: one is to modify the existing filesystem such that the deleted inodes have their `deleted' flag removed, and hope that the data just magically falls back into place. The other method, which is safer but slower, is to work out where the data lies in the partition and write it out into a new file. Sections 4 through 6 detail some steps you need to take before beginning to attempt your data recovery. Sections 7 through 10 discuss how to actually retrieve your files. 4. Unmounting the filesystem ============================= Regardless of which method you choose, the first step is to unmount the filesystem containing the deleted files. I strongly discourage any urges you may have to mess around on a mounted filesystem. This step should be performed *as soon as possible* after you realise that the files have been deleted. The simplest method is as follows: assuming the deleted files were in the /usr partition, say: # umount /usr You may, however, want to keep some things in /usr available. So remount it read-only: # mount -o ro,remount /usr If the deleted files were on the root partition, you'll need to add a `-n' option to prevent mount from trying to write to /etc/mtab: # mount -n -o ro,remount / Regardless of all this, it is possible that there will be another process using that filesystem (which will cause the unmount to fail with an error such as `Resource busy'). There is a program which will send a signal to any process using a given file or mount point: `fuser'. Try this for the /usr partition: # fuser -v -m /usr This lists the processes involved. Assuming none of them are vital, you can say # fuser -k -v -m /usr to send each process a SIGKILL (which is guaranteed to kill it), or for example, # fuser -k -TERM -v -m /usr to give each one a SIGTERM (which will normally make the process exit cleanly). 5. Preparing to change inodes directly ======================================= My advice? Don't do it this way. I really don't think it's wise to play with a filesystem at a low enough level for this to work. There are also has problems in that you can only reliably recover the first 12 blocks of each file. So if you have any long files to recover, you'll have to use the other method anyway. (Although see section 11, `Will this get easier in future?' for additional information.) If you feel you must do it this way, my advice is to copy the raw partition data to an image on a different partition, and then mount this using loopback: # cp /dev/hda5 /root/working # mount -t ext2 -o loop /root/working /mnt This does, however, require a recent version of `mount'. (Although you should get version 2.6 or newer anyway, as all earlier versions have a major security bug which allows peons to get root access. The major distributions, that is, Debian, RedHat and Slackware, have all been updated with version 2.6 of `mount'.) Using loopback means that when you completely destroy the filesystem (as you quite possibly will), all you have to do is copy the raw partition back and start again. 6. Preparing to write data elsewhere ===================================== You need to make sure you have a rescue partition somewhere. Hopefully, your system has several partitions on it: perhaps a root, a /usr, and a /home. With all these to choose from, you should have no problem: just create a new directory on one of these. If you have only a root partition, and store everything on that (like me, until I can get around to repartitioning), things are slightly more awkward. Perhaps you have an MS-DOG or Windoze partition you could use? Or you have the ramdisk driver in your kernel, maybe as a module? To use the ramdisk (assuming a kernel more recent than 1.3.48), say the following: # dd if=/dev/zero of=/dev/ram0 bs=1k count=2048 # mke2fs -v -m 0 /dev/ram0 2048 # mount -t ext2 /dev/ram0 /mnt This creates a 2MB ramdisk volume, and mounts it on /mnt. A short word of warning: if you use `kerneld' to automatically load and unload kernel modules, then don't unmount the ramdisk until you've copied any files from it onto non-volatile storage. Once you unmount it, `kerneld' assumes it can unload the module (after the usual waiting period), and once this happens, the memory gets re-used by other parts of the kernel, losing all the painstaking hours you just spent recovering your data. If you have any of the new `superfloppy' removable devices, they're probably a good choice for a rescue partition location. Otherwise, you'll just have to stick with floppies. The other thing you'll need is a program which can read the necessary data from the middle of the partition device. At a pinch, `dd' will do the job, but to read from, say, 600 MB into an 800 MB partition, `dd' insists on reading but ignoring the first 600 MB. This takes a not inconsiderable amount of time. My way round this was to write a program which will seek to the middle of the partition. It's called `fsgrab'; you can find it in source form in the following uuencoded, gzipped tar file, along with usage and installation instructions and a copyright message. To retrieve it, you should need only the following commands: $ uudecode < Ext2fs-Undeletion-mini-HOWTO $ gzip -dc fsgrab.tar.gz | tar xf - begin 644 fsgrab.tar.gz M'XL(`$-JR#("`^Q;WW/;1I+VZT[='S'EETA5#"/+L;V)GRB)LGDK40I)Q='C MD!B26(,`%P-(YG]_7W?/``.2OSM[=XZG9Q_>OGNEW[WZ M7_BI765*K5\94Q;Y=]?9TKWZ/_>S%/U?WMT_CL:?_IHSWIR=O??Z/J+_G]^> M?7@?]/_V_.T'/#U_4I/OS[+SU^ MI*]+:_6T6%;/IK3ZNJCSQ%38H*='^:*O](L_[S^\T[?&.3UXLCU]:3;S,DU6 M^.?M0)^=OWF+`QZF`Z6'3[;<%:`D=7IKRTU:53;15:$7($F;/-%)ZJHRG=>5 MU5@[Q_$;>IA:IW2QU-4:;V;IPN;.ZJ18U!N;5SV-]7JQ-ODJS5%J7>%*[2+G!/_R76I:M<**S,5WSX M;'9Z5]2E6D)42;&A)V[-ZT$\DP#FJK[6%SO0G5>E<:"OPEFL,)O;TF3ZOI[C M:'7C&0&Y:5[9/)&C5K4I#?ZV?)3^WE'T3`6:?_P12S9$IZNQC`YMV,$1M)89 MA5A`H]/L'WV21.I4ES0=2#/;;0;AT^$L']:![5J*:BWE!Q=),&=N3+[3!=XI M];8L$+DV^GE=T,YUM2Y*!REM8`=8J6HGZ@-))]-B8_UK+UEEA[E%`7.!^.8[ M%81]D\Y+4^[T"YREN:NL2?JG6C\6M5Z8G)G=:2&&1>\I=M!@4?3):KZL;:Z? M(=BM-5])&BS50$F/'A%%I5W:LB1V(`&OP![9I-J6.!\8H\+*P MZ1-M4I<+VCJ!9DH6V,K"URH57H31XL_H55KC+;5CC7@=QJ=!XT*HI$URG=MG MH3?(_:,84=CN:UX\-_LF!>WI:&?(V;%V9@6]6ME%):[#49HH&"N%)Q*FS=G5_2&R$Q%.)NV^RJ."M%*2XY;,H*SJJYF\TSD% M+NTR4_'F"UM6!@QCQ18/TWF:I57JXQ#M+!)51S4:2[)'%'GQ;XHD79+YLBBN M\\LF,'ZS6F%ID7GJK(\K#/`";1T.'6 M,`FLV01C0%:A&,2[BL'@7VFI@FK(A^TQ*X'=5VM=/4.GE=VZ7_7)FU/.2Y(J MNU*'6:J3\U/(#W[NS23*3,_K%$(E&3E^F-D5W)PSGN.,[%->+]8P]OR)TQ"K M,3Z/J1YD#A(B75A#&N/PB7CK6:%=R5G`D!@\>V,P>&]PB@5N0Q:NR7`!IO+$ M-:J0<)H7>+^D++3C(YF[3K*!(D;+@QS#Q*<.+9 M*A\M7&Q!(->K#,0\!^-@`PHYG4XLH)(T-UD/9PA+E&0@"*3V#>?2LDCJA9#! M282T"^ND#1":,U(]:2':2_E\]`,6;.N*,XR8RS4]SG8]/B0.3T12M0:D0.K& M64CW),L**82Y]\EQ2X\KRK.P.XJM'$&>BC3A\Q.*CJ5PC`06S($R(YS3B-"; MS$E,I'F2/J5)343I8LZ!1`YI\`P\/M<6MKE@;^,\M&ZWP6^D(0L4O>O[H`F; M('.!FMEX6.(;DQ"8T8O,&D\A1.`9$O>;-Q@J$=/TIO6#AQL4Y?$QR;U99QB8 M]0,&VY+^&\_E_%2`0XF:M"'+V[H2:UL(&E@6A/;ZZC_4GP)D/)T- M)[=3/1A?Z.'MW=7H>G0YH`^( M^+,^(Z=C4,F;(PL;'`B.>2[*KSXR$#*$VIPR)!K*O=O,>'LEHVC#SKK(*+DX ML_/0=@,$"JFW<2-1=9-_1(8!)Q^'%WT1^^M[H>\UT+.%X'J*,4M#/J>%B`>B MGN,>;/(ULS(WXLU\18AO&<[,\Z_B MTRG3`LYQK*SU8O/FW-E9;XN2S8#!1$]Y`IH:@CB@^!Z;C`LAM\G-"<4.XI\U MIC+X9FU6)+*3SXB,"`1+B+C7O$`',GA?9#6!=SJBJ,G6`6G]XUP%S>C7\>FO M"7D.*91[S^`09Y($H(#=Q.G7R!VOX2@#A/&W@4 MX2]*NM]"91Y`,UO.>6LY'M_QCL)5>=Q@0L3TD4U)9,.*FO/B1LA],13W?"X5 M.XV!)H?V;B#T`5X?2253S]P;9>8%M7<.[!*F`<"]L5:,1+AP-LKCORIN'IG3 MM@A8F-I)!=%@QF6:2?I<0+8L6/!([NU-CO=P%%?9IT.-R?*6F",[A`B44+7E M#4]6]86.^0$=;)LD@&;;2%X0COPR[BR4501U$A1AA_"F&I!#LF5(4>V-:)LTN9$`O(8&0^H7]Q6F` M[HWH0Z+/85>,*X%J$^G-<'5`[:G24!I"G/',(]`BP$8UH8B2;)0?4I>64FJ( MPN019'K\>K0A@\0T]P11CZE,D&E+BA9<&(*ZE()\24H!4"*#%GO*\Z)&=*$F MH$_"[!2=B*>/1CS#&_@/7JY]3@C3HG[I!036V(?W`J&C>>&T;5AP=XT]/H+U M8O%!VJPNWF'?87P:M5D6\A=MI[G8+?13:I_W8B+OTB*\D^&WA>5P]2LEV$[* MKIS-EJ'G&'0`VG@+RG6VT#&3U2'5>0) ML0^*9@GQL(OVJ(_%O>J6SP*IK2&?76DO('&_P[B]HZGI7%?-"VK/Z)S91%+! MVQQZN,:4$".E2>HZ247M)Q4.K#'@]$E+]@A%H7\K1"'5E8`T@-MVB-1Y`@(" M&$8-\8U:XE[UBE1;^F,"R*PY6T@[!!]P\2ELE79ER@3)@/6/E_0SI6EICLWP M8B^Z)B!*N?]>-0'3RXF3$0&CJ/_'0-55*FX=89E4=R7=:``%,+'2","ZCQI: M6G/AT![%Y8VRWVPIY6]HG$EOB%H8V5%A1P5440+.9=3-".64.PH%P/,HI](B ME=N<#44ZLUJ1E,*VON81/D@JQS92^UB+`R1_^!TDH>EU5 ME"BL?$QO^1/LVT:A>1GB7T2=A$VV::I2CF:YM]^'ZOLL[%-/):0DTP!_SD\I M1Q7S?U)/)?3`H;U%77&\(41V)/^J:?"X-TS#N684]1*(0C"@EIGW*6EI0`(M M?AHLD).W!%=@OXTVZ+/,YX/71DV% M[R!!R35==EC!7GD+[%9L3)G"_NO0&&J;A)1T!(U]A`A[#2([Y,PT_L20NZ>? M3);*=I!9ANA<4#N$51.UUG2@,[E0H^!D;_A M"A4"93];!JSM!1?;:X^SL,B>=]B7>)2C]Y73T0,#/TG`_YX.7I:_M*O=_0%V.M/W=>YK"HHY-DSEI$5W&^4YMMS*#//E&])J+AIAHZ<@U MKB^G*SY=C@SW,0=TX7,<4E.ME+95"RJ[K'9HLOU[B<)ENY4:9$K8*^8N(2WV?C&$/=4(7'Z&XI](Z(3ME-M: MUK@-8+9WP$_L+GS%1UG#]^/H-H\O!YM63P-JX]=.J&R7=J'?&3*:&0T6 M+Q[_\ND4/^E>26R0[XI,36F@\LF,LDA*BNS<>QXA2S5^&`1,$)HOA9K@*CTK M%D:X9F?U4H8(""WJ"3;W;V%R(2W;\9N&,'8=5A.5-Q2+`P&H!^FB"_];UIE$ MEBPU*!X9[KT3U87R+JXVR22WU5X-YE)J2H;+:38=/V[!P;9AGT`QFSC=8:ZH MQ)>V;?W][Y0 M/.1']D%R*>AN>KZ3BS'N5W0`]E[S_L0W&%]LLI]*-UT+<*W'\J,/[OMRC5.G&>H#R/:C_)QQWAAKV M',@;/Y7(P1M#2%/A(MD_D4D1<>)N+S&ZX`]TP;LY%E5TG6U?N`P-(Q0^/*7( M#+YSN:Q+OJ_J#)SX&JQMJO^@FV+3!U3_(2*H"14MOC_ M!>FI]4!_I12%8^9CKR+[T->CI21V;J?`19N;`4H"J-K_621^C3D:1KU@YM4'NDW1^SY2$^NV\27*5U0\Y/.;5+C MR^_*R(6??Z+7XYY^X=&XHZD=F)=+-W4&-[5R6207&,@A*X\KVZBOXFN;:%K/ M0I?\ZG_0(D$O8-AON![_MK_<#+)!.TVTS-%G0F0DQE1718[E`F['WFD M('+N"">$4Q#\!/86/(93-!=L_HHE05I8T(@&M^V;OU!&,JH`'\(B1QXN+/S( M)QD#J`KBG4-(!)ZE$17G.5XVIV!(-^HE):VF'<1*_@[Y@N&B2Y^#AA3^N;89 M(6DIAFF2+A>GM(SR)/7R%N2,BSHSB+1IN:@WCJ.V1+BYR=H0;N/MHTE4)4W) M<)\2%D77$GN3JWZ`,A<34O&Q=(,ZZK3Q(SPV:J7U^YK_$ZZ/I$]>. M55"C'Z:Z\]TS;M>%03W?JY/&05KM_&V0XFZVK/S8/7QM?$5#W$44AEL^/TE# M3*]*OV,8PVP+[(Z*!?3WFOZJ2LGT*9)(BM_*>$:P_BVWY$E@6M^R'FV!]>U( MCEK17`?<6J*./Z8IQ9_I"K_D.TB:[CL@R28J6#N'+E^3\#2BC^=%+@UOQX&3 MYUH64'J+Z*2ER44""[)/P9"F/6FFW9ILA,,CIO=,L M:&@-]+7!R!,IXR?-O(0/@SX32B!>%RECPMF>U\1FRB-Q1"B=0MU]'G!Z]D7B M'&*P3^(`AX[`,X(ANDD3Z#F0$4/?* MTO+MFF_0.RQ&0R_(:W(7IR00-ZST9#335-U7.U\'D'9.SB!@@U)`M8*0T%$[ M?X!-*"7FF+TCZE?'LK*J>AYB?Y(H937O[#S MD&$BPZ<4W$&[VZ8ECZV'-I,CQ_5OR-;N2W;^=!0&W,W9\G5^M[: M@T)"0F4T4.WNA(4+8'1/!WB79KIEA*0+.#Z]0;7J>0G7^_J[CP7N@6BR-[XC)Q6PW/R@_0T_P MO:VD/31LLD!S'QF'N3^1_-YQ+_GK1_X*1[&QY&1.<3YHFHRNF7CV7].@),9R MYQX&/`\FG[2TT,CXJC`9>S?[7OD4S$Y@`4).+>.\>+]M`O!'X1L^G>_-R$[% MIFAJ=OKFC\PV)`@P/HTTKZPDGF2[]JM.XSO]93"9#,:S1];_F[Z^&%X.'J9# M/?L\U/>3NT^3P:T>3<-4[)6^G@R'^NY:7WX>3#X->[1N,J05\5XT(QMM@%5W M_/?PC]EP/-/WP\GM:#;#;A>/>G!_C\T'%S=#?3/X`FD._[@$,:Q)V,/GV>Z<]W-U?#"4_K_H33^45]/YC, M1L.I`AV_CZZZ3+T>3$'V:_UE-/M\]S!KB"?F!N-'_8_1^*JGAR/>:/C'_60X M!?\*>X]N0?$0#T?CRYN'*QX$OL`.X[L9Y`3.0.?LCD43UH;=00SV5[?#">0W MG@TN1CCV1A'\'SQ0"B_?+@9@(F'R?W==$C]&Q(A-H'`)Z/I/_1@ MJKQ@?WL8-!M!NMCC=C"^9$7M*9+8U8]W#Y0UP/?-%2U080$):JBOAM?#R]GH M=Z@7*W',].%VZ.4]G;&`;F[T>'@)>@>31ST=3GX?79(K&3X.]G`P_B&N)T,?WL`/TQ';QX@LYG<[EH%C**USL'%'?AM.>:HR`C_;#Y#T]O1]>CN@?>`[3@ZYO1"KPHM\>2(OXP&^B!U`G ML49VZ%5&/DBV-@XV@K/W_?*D/7O/_L@N;NZF9&PX9#;03#%^7PQI]60XAKS8 MG0:7EP\3N!:MH#=`S?0!SC8:LU(4\/)IOKP>CF87)@8SCY#B*D M+=G6&H4$(YN>]M@&].@:1UU^]MK3':]]U)^ABHLAE@VN?A]1Y)%S%'QA.O(R MN?,[>#ER8./OFH(_7G]D@)]F_P=;&LY)O_U*35S*`P.N2:7/.F,4@`\?*>R. M`7E\KG-DQSX_)DBO6;%%BO:8J)VFC+[EYF?U?,I<\;=`7*50B4BSK'9-%I(" MS]?=5#A02X$[TVLJ-`3ZR+0[9Z*T4MV,()FP^=H.C2=U6IS1%T*;*^/01`S? MBPN-V:HR_N*I!4C-2&\17YD2BN&"R)DEL484-V]OPF*>\N.;)GKB;UKH?K#Y MRJA\#T4F!P$3GNS.WUP!PCL/UOZKO6MO3MO8XO??[*?80G(M,8"QX\13,)XZ MF"2^<4W&=MKF]F:(`&$8@\0@<$-3W\]^SVNE%<:OQ$WO=+23&0MI]YQ]G\?^ MSB:!'!.0!TD1C6A`[A12[\R9/VGRN5@IR(%6'XCS2D]"LH,(D$-X/FKHG(\> M*,81I3MTDD`A=[`_J;S!#5@=L`8J&YY3,>D.6"!]#8+?8TB11[.`L.&[1&LY MK'JQ`/I$`"4_*3Z[S)5L4RN(*#7:M3C",37&K`$G`6*,HIRMAGRNBC9.T-E1 M2G>,$7O7*TM),`6'DQLFA\F!&%%QTDAI]ZH.75[=`?9QK)AB`T3VS*27C>(% MBPH&L\B0$3!JC(#'C<@(^5H3B'1%JT,`Z0=E&$LNR&CKW#J+ZQ*=9 MHN(@HVN,.1HJBN1%6RN2IJ-SW9[5"9@BA16YGK!`)*RCS*0O:VC3PDR_20VF M\LNQ_<4[AO!C,`KBE-`K8,-"T'/&VRVA"3BR$A5D']%ITS"`^G,((.C[L,L- M1^SJ3"$T4HC4HMD+32")A]TVC4&\H^$Y[YR*`(^0CW:BB*,H4MA66#"^(*A> M!:!47[`V;Z;SI"8T;T7)ZU#T#8.W]N:_8FES8`H2.Q4["\R)EC-9M==LRM29JC* M8#%!PX[.M1*4MZD?U2$N+;/51-JFHDE2=N.U\6:M/AVER.E'PH^.BB/T:B[0 MH8%G;'0"#'89>12L8*>559/8)?;,TVKO^&H<`LE2%VIP3HZ,L1_,HY)#+%Z!K#O@HP-#[B",W7@DSW$#TX>TP=`2#O6&6G"Y@I87! M;E%O@%XV'8[H]A%44/A#$6_HB(8FINLGF$'BR;UFDXW]*G)2E/@T M#&5%OL:7#,3':E-[*_+P4'8:XIDT;C9TE43LE%$&#TX1F;C)LV2BXT:N"2@5 MA.:R.5J>]"C&H2@A;IQ&O"G\9F"A)HR[!\J;B9A9<;N%6GV[Q55GYC^R]/>X M_XG_E+M_WOU/E6OO_WJ^M?W4W/^T5=FNH`]@:VL[N__I6Z3U@C9CKTLEDE*= M48C2CO%Z@7;\3[--EU&:,>+PAM$*HP@)W=TN MNL4F0F)ILX@;_M5VD643(;W[FD4WF$34D3=;1;JPKE1><,=Z9Q[`L/7*@UWK M':R(=53+HZ77L]XP3+_J=X/9Z$HN4!.NO/.F9_A.45!L5].E(GUO!GWC4!@] MA3OH0G\,32^7RZYNM[V9S*=V6SL.6W#:H9"\/BS*HMYT$?8X]4%G"URWEJ)= MP$@11%"'7>U$P]_]-MACD$=AK-8T$';2Z^UA'-R M\._F!_C9U8W6NZ-3?(STR9N#M_"TW_SIH-'\\)\@9PB(M`0"I8'UNM2I)M3X M$AD\S?U=L'#6MDV-J2'RWIN/.,X,VV33ZE:Y*GQ0-F?%6;9F,AN]WA(!NW14 MI=I?4QB')%VX8A<>5"64D];:P!]-P-B*L-.L7-PK'`Y-8RB5(KF1)AX/+10W M@W#C7"K>_!DFYJ5"J(XBZ+!#4:?3LVY1)E4!?ER824`1QA-8)_U>C5Z$_7X; M[P.<@]A"&ZX=^?YY+BSG6JW6I4'>1<5W]7U\W62U<]BF!O`^F+Q>#79_4((0]Z MK;-6Y_-7_OI!.A_S#R/8!A=X MPQ=DE@TS]P06`^P`*"#)_A-0^@AGHX;LY!#"(D*HWP,&((MA9C&_HFZUC_=; M1X?OW5K,JH_5JEALR%L1K,VXJ-04>3\9$R=^8YC(O`-.#DU(%W8IW"@+VAY6 M$]B'&E9=CZ@$<(ZG;5&?-)MOVB?-4ZMFDA]JI__XPY2&R29EEGL&:?9!8ON] MZZM*"[M.*\U%$80[RHJ:PDJ`7+8LD;(F@UE'D(O^4%N@5%$G&4TSXKS2D/@W M-X5RIYO"V\]-33$>C;HH#,Z&L#?$+?XFK[`W/X%[G#EAS^12O`U/EK@H_R[3 M`KV@5DI=V; M&S1G!%L*?WC!IT/.A0?[ZG@F3);EK:91R=V^FQ/1*Z6I'A>>^>ZU,6KN<^J@I_P?W/>O/IUF9R__?6!MG_VQN9_?\M4IY\L70\ MBHYCG@XJK_*WF?68Y:&,^OS=;'ID^5`6??XA#?K\`]KS^57F_*J6?XTQGW\@ M6S[_@*9\?LF2QS8_J!V??U`S/G_KV>;;X^;+@U_JZ_-HNHX2'!SM'QS7 M'SO\R5WO#`.E8!B.ZJ6?\4"E]+/:;[YX]ZI>:FVJ1O.7T^.]NFJ\/-Q[=5)_ M_)D^7>K'G[$$_N4B.IE M\E$*Q-G#.'LHKY(O"D'N`2C4I>E8E_JZ\%]=*(?:*^,DZN(PFEK@+.7,!(P/ MT(2BL.TJZ@WJT9G\U&#ZEL8:_Y\(X`?]HDMG]$=8/W:X\URERF]?MX[>"ST= M,^";R)F:*I^\>PD]W#RI9F<4=Y'_Q\V]_1^;?Q*/6^3_QK/-;9'_SYX^JSQ' M?.GV9B63_]]N_._O^E>WJ`!YZ"%^Y-BZ M)TS5K<3F1\$<>=`S"AS<<9-?J$4I14X^)<6D:^_C[C-E2@.EOL37I52+4>-Y5<]+$T-*<$36%P#^=:8V5H:*(SP2*% MT4WT2YQ*^W:P`< lsdel.out Now, based only on the deletion time, the size, the type, and the numerical permissions and owner, you must work out which of these deleted inodes are the ones you want. With luck, you'll be able to spot them because they're the big bunch you deleted about five minutes ago. Otherwise, trawl through that list carefully. I suggest that if possible, you print out the list of the inodes you want to recover. It will make life a lot easier. 8. Obtaining details of the inodes =================================== Debugfs has a `stat' command which prints details about an inode. Issue the command for each inode in your recovery list. For example, if you're interested in inode number 148003, try this: debugfs: stat <148003> Inode: 148003 Type: regular Mode: 0644 Flags: 0x0 Version: 1 User: 503 Group: 100 Size: 6065 File ACL: 0 Directory ACL: 0 Links: 0 Blockcount: 12 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x31a9a574 -- Mon May 27 13:52:04 1996 atime: 0x31a21dd1 -- Tue May 21 20:47:29 1996 mtime: 0x313bf4d7 -- Tue Mar 5 08:01:27 1996 dtime: 0x31a9a574 -- Mon May 27 13:52:04 1996 BLOCKS: 594810 594811 594814 594815 594816 594817 TOTAL: 6 If you have a lot of files to recover, you'll want to automate this. Assuming that your `lsdel' list of inodes to recover in is in `list', try this: # cut -c1-6 list | grep "[0-9]" | tr -d " " > inodes We save this file `inodes' containing just the numbers of the inodes to recover, one per line, because it will very likely come in handy later on. Then you just say: # sed 's/^.*$/stat <\0>/' inodes | debugfs /dev/hda5 > stats and `stats' contains the output of all the `stat' commands. 9. Recovering data blocks ========================== This part is either very easy or distinctly less so. If the file you are trying to recover is not more than 12 blocks long, then the block numbers of all its data are stored in the inode: you can read them directly out of the `stat' output for the inode. Using `fsgrab' to write the data out (possibly in more than one section, if the file was fragmented) is then a straightforward task. To take the example we had before, repeated here: debugfs: stat <148003> Inode: 148003 Type: regular Mode: 0644 Flags: 0x0 Version: 1 User: 503 Group: 100 Size: 6065 File ACL: 0 Directory ACL: 0 Links: 0 Blockcount: 12 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x31a9a574 -- Mon May 27 13:52:04 1996 atime: 0x31a21dd1 -- Tue May 21 20:47:29 1996 mtime: 0x313bf4d7 -- Tue Mar 5 08:01:27 1996 dtime: 0x31a9a574 -- Mon May 27 13:52:04 1996 BLOCKS: 594810 594811 594814 594815 594816 594817 TOTAL: 6 This file has six blocks. To write them in the correct order to a new file, say `recovered.001', say: # fsgrab -c 2 -s 594810 /dev/hda5 > recovered.001 # fsgrab -c 4 -s 594817 /dev/hda5 >> recovered.001 There will be some garbage at the end of `recovered.001', but that's fairly unimportant. Of course, it is possible that one or more of the relevant blocks has been overwritten. If so, then you're out of luck: that block is gone forever. (But just imagine if you'd unmounted sooner!) The problems appear when the file has more than 12 data blocks. It pays here to know a little of how Unix filesystems are structured. The file's data is stored in units called `blocks'. These blocks may be numbered sequentially. A file also has an `inode', which is the place where information such as owner, permissions, and type are kept. Like blocks, inodes are numbered sequentially, although they have a different sequence. A directory entry consists of the name of the file and an inode number. As well as permissions information and such-like, the inode also holds the location of the file's data. The numbers of the first 12 data blocks are stored in the inode itself; these are direct blocks. Then the inode contains the number of an `indirect block' (which points to a block containing the numbers of additional direct blocks). The inode also has the number of a doubly indirect block (which contains a list of block numbers of indirect blocks) and the number of a triply indirect block (which contains a list of block numbers of doubly indirect blocks). Read that again: I know it's complex, but it's also important. Now, the current kernel implementation (certainly for all versions up to and including 2.0.27) unfortunately zeroes all indirect blocks on file deletion. So if your file was longer than 12 blocks, you have no guarantee of even being able to find the numbers of all the blocks you need. The only method I have been able to find thus far is to assume that the file was not fragmented: if it was, then you're in trouble. If your file needed an indirect block, then the unfragmented layout will be as follows: #Blocks Contents 12 Data (direct blocks) 1 Indirect block 256 Data (blocks in the indirect block) 1 Doubly indirect block [1 Indirect block of the doubly indirect block 256 Data ] (256 times -- one for each indirect block in the doubly indirect) 1 Triply indirect block [[1 Doubly indirect block 256 Data ] (256 times -- one for each indirect block in the doubly indirect block of the triply indirect block) ] (256 times -- one for each doubly indirect block in the triply indirect block) I hope that's clear. If your file used no more than 268 blocks, then it's entirely feasible: there are 12 data blocks (whose numbers appear in the inode itself), followed by an indirect block (whose contents have been zeroed), followed by another 256 data blocks. If your file was longer than that, then you add a doubly indirect block; after this, for each additional 256 data blocks you add one indirect block and 256 data blocks, a maximum of 256 times. If your file was longer than that, then the chances of it having been unfragmented is in general, I would guess, fairly small, but it should be apparent how the scheme is extended to longer files. You should note that I assume throughout that your blocksize is 1024 bytes, as this is the standard value. If your blocks are bigger, some of the numbers above will change. Specifically: since each block number is 4 bytes long, blocksize/4 is the number of block numbers that can be stored in each indirect block. So every time the number 256 appears in the table, replace it with blocksize/4. 10. Modifying inodes directly ============================= This method is, on the surface, much easier. However, as mentioned above, it cannot cope with files longer than 12 blocks. For each inode you want to recover, you must set the usage count to one, and set the deletion time to zero. This is done with the `mi' (modify inode) command in `debugfs'. Some sample output, modifying inode 148003 from above: debugfs: mi <148003> Mode [0100644] User ID [503] Group ID [100] Size [6065] Creation time [833201524] Modification time [832708049] Access time [826012887] Deletion time [833201524] 0 Link count [0] 1 Block count [12] File flags [0x0] Reserved1 [0] File acl [0] Directory acl [0] Fragment address [0] Fragment number [0] Fragment size [0] Direct Block #0 [594810] Direct Block #1 [594811] Direct Block #2 [594814] Direct Block #3 [594815] Direct Block #4 [594816] Direct Block #5 [594817] Direct Block #6 [0] Direct Block #7 [0] Direct Block #8 [0] Direct Block #9 [0] Direct Block #10 [0] Direct Block #11 [0] Indirect Block [0] Double Indirect Block [0] Triple Indirect Block [0] That is, I set the deletion time to 0 and the link count to 1 and just pressed return for each of the other fields. Granted, this is a little unwieldy if you have a lot of files to recover, but I think you can cope. If you'd wanted chrome, you'd have used a graphical `operating system' with a pretty `Recycle Bin'. [By the way: the `mi' output refers to a `Creation time' field in the inode. This is a lie! Or misleading at best. The fact of the matter is that you cannot tell on a Unix filesystem when a file was created. The `st_ctime' member of `struct stat' refers to the `inode change time' -- that is, the last time when any inode details were changed. Here endeth today's lesson.] Note that more recent versions of debugfs than the one I'm using probably do not include some of the fields in the listing above (specifically, `Reserved1' and (some of?) the fragment fields). Once you've modified the inodes, you can quit `debugfs' and say: # e2fsck -f /dev/hda5 You will get some informative output, and some questions about what damage to repair. Answer `yes' to everything that refers to `summary information' or to the inodes you've changed. Anything else I leave up to you, although it's usually a good idea to say `yes' to all the questions. When `e2fsck' finishes, you can remount the filesystem. The deleted files have now been placed in the /lost+found directory of that filesystem (so if the partition is mounted on /usr, look in /usr/lost+found). They are named according to their inode number; all you still have to do is work out their names from their contents, and return them to their correct place in the filesystem tree. Actually, there's an alternative to having `e2fsck' leave the files in /lost+found: you can use `debugfs' to create a link in the filesystem to the inode. Use the `debugfs' `link' command after you've modified the inode: debugfs: link <148003> foo.txt This creates a file called `foo.txt' in what `debugfs' thinks is the current directory; `foo.txt' will be your file. You'll still need to run `e2fsck' to fix the summary information and block counts and so on. 11. Will this get easier in future? =================================== Yes. In fact, I believe it already has. I hear that recent kernels in the development 2.1.x series do not zero indirect blocks. At the beginning of December 1996 (the last time I had chance to follow the linux-kernel list), there was talk of producing another 2.0.x production kernel that also leaves indirect blocks intact on deletion. Once Linus and the other kernel hackers overcome this limitation, a lot of my objections to the technique of modifying inodes by hand will disappear. 12. Are there any tools to automate this process? ================================================= As it happens, there are. Unfortunately, I believe that they suffer from the same problem as the manual inode modification technique: indirect blocks are unrecoverable. However, given the likelihood that this will shortly no longer be a problem, it's well worth looking these programs out now. Someone on the net mentioned `lde' by Scott D. Heavner. To be honest, I wouldn't recommend this as a tool for automating file recovery. It's more like a full-screen `debugfs' than anything else, although it does have some features like the ability to scan for certain types of file or for certain file contents. It also works with the xia (does anyone actually use this any more?) and minix filesystems, which I guess is its major selling point these days. URL: ftp://sunsite.unc.edu/pub/Linux/system/Filesystems/lde-2.2.tar.gz (although it's possible there's a more recent version than that -- I found that on a 14-month-old CD-ROM archive). It sounds like the one that *really* works is the GNU Midnight Commander, `mc'. This is a full-screen file management tool, based AFAIK on a certain MS-DOG full-screen file management tool commonly known as `NC'. `mc' supports the mouse on the Linux console and in an xterm, and provides virtual filesystems which allow tricks like `cd'ing to a tarfile. Among its virtual filesystems is one for ext2 undeletion. It all sounds very handy, although I must admit I've never used the program myself -- I prefer good old-fashioned shell commands. Apparently one must configure the program with the `--with-ext2undel' option; you'll also need the development libraries and include files that come with the `e2fsprogs' package. I gather that once the program is built, you can tell it to `cd undel:/dev/hda5', and get a `directory listing' of deleted files. The latest non-development version is probably 3.2.11; as with the kernel itself, development versions are *not* recommended to non-hackers. It is available from one of the following URLs: ftp://sunsite.unc.edu/pub/Linux/utils/file/managers/ (may be an older version) ftp://ftp.nuclecu.unam.mx/linux/local/ ftp://ftp.nuclecu.unam.mx/linux/local/devel/ (unsupported, developmental, alpha versions) ftp://ftp.cvut.cz/pub/mc/ (European mirror of public and alpha releases) Also try the website: http://stekt.oulu.fi/~jtklehto/mc/ I get the impression that the `mc' developers are getting quite excited about their next public release; this one will probably also support any effort in the kernel to avoid zeroing indirect blocks. A. Colophon ============ I intend to produce regular updates to this document as long as I have both time and something interesting to say. This means that I am eager to hear comments from readers. Could my writing be clearer?. Can you think of something that would make matters easier? Is there some new tool that does it all automatically? Who *did* kill JFK? Whatever. If you have something to say, about this document or any other subject, drop me a line on . B. Credits =========== "If I have seen farther than others, it is because I was standing on the shoulders of giants." -- Isaac Newton Much of this mini-Howto was derived from a posting in comp.os.linux.misc by Robin Glover . I would like to thank Robin for graciously allowing me to rework his ideas into this mini-Howto. C. Bibliography ================ Frisch, Æleen (1995), `Essential System Administration', second edition, O'Reilly and Associates, Inc., ISBN: 1-56592-127-5. Glover, Robin (31 Jan 1996), `HOW-TO : undelete linux files (ext2fs/debugfs)', comp.os.linux.misc Usenet posting. Peek, Jerry, Tim O'Reilly, Mike Loukides et al (1993), `Unix Power Tools', O'Reilly and Associates, Inc./Random House, Inc., ISBN: 0-679-79073-X. D. Legalities ============== MS-DOG and Windoze are not trademarks. Any resemblance these names may bear to those of any commercial software products is entirely coincidental. Unix is not a trademark, and has not been for quite some time now. Wake up and smell the flowers! The trademark status of the name `Linux' is currently being contested by lawyers, as a certain Walter R. Della Croce has made an apparently false trademark registration for the term. I however refuse to acknowledge the name as anything other than public property of all net.citizens. If Linus had wanted a trademark on the name, he'd have taken one out himself. This document is Copyright © 1997 Aaron Crane . It may be freely redistributed in its entirety, including the whole of this copyright notice, but may not changed without permission from the author or from the Linux Documentation Project Co-ordinator. Dispensation is granted for copying small verbatim portions for the purposes of reviews or for quoting: in these circumstances, sections may be reproduced in the presence of an appropriate citation but without this copyright notice. The author requests but does not require that parties intending to sell copies of this document, whether on computer-readable media or on paper, inform him or the Linux HOWTO Co-ordinator of their intentions. The Linux HOWTO Co-ordinator is Greg Hankins .