diff --git a/ChangeLog b/ChangeLog index 567534d..dab4732 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,746 +1,747 @@ LibPST 0.6.38 (2009-xx-xx) =============================== * add python module interface to the shared library for easy scripting. * the shared library must never write to stdout or stderr. * fix pst_attach_to_mem so the caller does not need to initialize the buffer pointer. + * remove readpst -C switch, obsolete debugging code. LibPST 0.6.37 (2009-04-17) =============================== * add pst_attach_to_mem() back into the shared library interface. * improve developer documentation. * fix memory leak caught by valgrind. LibPST 0.6.36 (2009-04-14) =============================== * spec file cleanup with multiple sub packages. * add doxygen devel-doc documentation for the shared library. * switch back to fully versioned subpackage dependencies. * more cleanup on external names in the shared object file. LibPST 0.6.35 (2009-04-08) =============================== * fix bug where we failed to pickup the last extended attribute. * patch from Emmanuel Andry to fix potential security bug in pst2dii with printf(err). * properly add trailing mime boundary in all modes. * move version-info into main configure.in, and set it properly * prefix all external symbols in the shared library with pst_ to avoid symbol clashes with other shared libraries. * new debianization from hggdh. * build separate libpst, libpst-libs, libpst-devel rpms. * remove many functions from the interface by making them static. LibPST 0.6.34 (2009-03-19) =============================== * improve consistency checking when fetching items from the pst file. * avoid putting mixed item types into the same output folder. LibPST 0.6.33 (2009-03-17) =============================== * fix fedora 11 type mismatch warning (actually an error in this case). * fix large file support, some sytems require config.h to be included earlier in the compilation. * compensate for iconv conversion to utf-7 that produces strings that are not null terminated. * don't produce empty attachment files in separate mode. LibPST 0.6.32 (2009-03-14) =============================== * fix ppc64 compile error. LibPST 0.6.31 (2009-03-14) =============================== * bump version for fedora cvs tagging mistake. LibPST 0.6.30 (2009-03-14) =============================== * improve documentation of .pst format. * remove decrypt option from getidblock - we always decrypt. * rename some structure fields to reflect our better understanding of the pst format. * track character set individually for each mapi element, since some could be unicode (therefore utf8) and others sbcs with character set specified by the mapi object. remove charset option from pst2ldif since we get that from each object now. * more code cleanup. * use AM_ICONV for better portability of the library location. * structure renaming to be more specific. * improve internal doxygen documentation. * avoid emitting bogus empty email messages into contacts and calendar files. LibPST 0.6.29 (2009-02-24) =============================== * fix for 64bit on Fedora 11 LibPST 0.6.28 (2009-02-24) =============================== * add X-libpst-forensic-* headers to capture items of interest that are not used by normal mail clients. * improve decoding of multipart/report and message/rfc822 mime types. * improve character set handling - don't try to convert utf-8 to single byte for fields that were not originally unicode. if the conversion fails, leave the data in utf-8. * fix embedded rfc822 messages with attachments. LibPST 0.6.27 (2009-02-07) =============================== * fix for const correctness on Fedora 11 LibPST 0.6.26 (2009-02-07) =============================== * patch from Fridrich Strba for building on mingw and general cleanup of autoconf files * add processing for pst files of type 0x0f * start adding support for properly building and installing libpst.so and the header files required to use it. * remove version.h since the version number is now in config.h * more const correctness issues regarding getopt() * consistent ordering of our include files. all system includes protected by ifdef HAVE_ from autoconf. * strip and regenerate all MIME headers to avoid duplicates. problem found by Michael Watson on Mac OSX. * do a better job of making unique MIME boundaries. * only use base64 coding when strictly necessary. * more cleanup of #include files. common.h is the only file allowed to include system .h files unprotected by autoconf HAVE_ symbols. define.h is the only other file allowed to include system .h files. define.h is never installed; common.h is installed if we are building the shared library. * recover dropped pragma pack line, use int64_t rather than off_t to avoid forcing users of the shared library to enable large file support. * add pragma packing support for sun compilers. * fix initial from header in mbox format. * start moving to PST_LE_GET* rather than LE*_CPU macros so we can eventually remove the pragma packing. * patch from Fridrich Strba, some systems need extra library for regex. LibPST 0.6.25 (2009-01-16) =============================== * improve handling of content-type charset values in mime parts LibPST 0.6.24 (2008-12-11) =============================== * patch from Chris Eagle to build on cygwin LibPST 0.6.23 (2008-12-04) =============================== * bump version to avoid cvs tagging mistake in fedora LibPST 0.6.22 (2008-11-28) =============================== * patch from David Cuadrado to process emails with type PST_TYPE_OTHER * base64_encode_multiple() may insert newline, needs larger malloc * subject lines shorter than 2 bytes could segfault LibPST 0.6.21 (2008-10-21) =============================== * fix title bug with old schema in pst2ldif. * also escape commas in distinguished names per rfc4514. LibPST 0.6.20 (2008-10-09) =============================== * add configure option --enable-dii=no to remove dependency on libgd. * many fixes in pst2ldif by Robert Harris. * add -D option to include deleted items, from Justin Greer * fix from Justin Greer to add missing email headers * fix from Justin Greer for my_stristr() * fix for orphan children when building descriptor tree * avoid writing uninitialized data to debug log file * remove unreachable code * create dummy top-of-folder descriptor if needed for corrupt pst files LibPST 0.6.19 (2008-09-14) =============================== * Fix base64 encoding that could create long lines * Initial work on a .so shared library from Bharath Acharya. LibPST 0.6.18 (2008-08-28) =============================== * Fixes for iconv on Mac from Justin Greer. LibPST 0.6.17 (2008-08-05) =============================== * More fixes for 32/64 bit portability on big endian ppc. LibPST 0.6.16 (2008-08-05) =============================== * Use inttypes.h for portable printing of 64 bit items. LibPST 0.6.15 (2008-07-30) =============================== * Patch from Robert Simpson for file handle leak in error case. * Fix for missing length on lz decompression, bug found by Chris White. LibPST 0.6.14 (2008-06-15) =============================== * Fix my mistake in debian packaging. LibPST 0.6.13 (2008-06-13) =============================== * Patch from Robert Simpson for encryption type 2. * Fix the order of testing item types to avoid claiming there are multiple message stores. LibPST 0.6.12 (2008-06-10) =============================== * Patch from Joachim Metz for debian packaging, and fix for incorrect length on lz decompression. LibPST 0.6.11 (2008-06-03) =============================== * Use ftello/fseeko to properly handle large files. * Document and properly use datasize field in b5 blocks. * Fix some MSVC compile issues and collect MSVC dependencies into one place. LibPST 0.6.10 (2008-05-29) =============================== * Patch from Robert Simpson fix doubly-linked list in the cache_ptr code, and allow arrays of unicode strings (without converting them). * More changes for Fedora packaging (#434727) * Fixes for const correctness. LibPST 0.6.9 (2008-05-16) =============================== * Patch from Joachim Metz for 64 bit compile. * Signed/unsigned cleanup from 'CFLAGS=-Wextra ./configure'. * Reindent vbuf.c to make it readable. * Fix pst format documentation for 8 byte backpointers. LibPST 0.6.8 (2008-03-05) =============================== * Initial version of pst2dii to convert to Summation dii load file format. * Changes for Fedora packaging (#434727) LibPST 0.6.7 (2008-02-16) =============================== * Work around bogus 7c.b5 blocks in some messages that have been read. They appear to have attachments, but of some unknown format. Before the message was read, it did not have any attachments. * Use autoscan to cleanup our autoconf system. * Use autoconf to detect when we need to use our XGetopt files and other header files. * More fields, including BCC. * Fix missing LE32_CPU byte swapping for FILETIME types. LibPST 0.6.6 (2008-01-31) =============================== * More code cleanup, removing unnecessary null terminations on binary buffers. All pst file reads now go thru one function. Logging all pst reads to detect cases where we read the same data multiple times - discovers node sizes are actually 512 bytes. * Switch from cvs to mercurial source control. LibPST 0.6.5 (2008-01-22) =============================== * More code cleanup, removing obsolete code. All the boolean flags of type 0xb have length 4, so these are all 32 bits in the file. Libpst treats them all as 16 bits, but at least we are consistent. * More fields decoded - for example, see We should be able to use that data for much more complete decoding. * Move the rpm group to Applications/Productivity consistent with Evolution. LibPST 0.6.4 (2008-01-19) =============================== * More fixes for Outlook 2003 64 bit parsing. We observed cases of compressed RTF bodies (type 0x1009) with zero length. * Document type 0x0101 descriptor blocks and process them. * Fix large file support - we need to include config.h before any standard headers. * Merge following changes from svn snapshot from Alioth: * Add new fields to appointment for recurring events (SourceForge #304198) * Map IPM.Task items to PST_TYPE_TASK. * Applied patch to remove compiler warnings, thanks! (SourceForge #304314) * Fix crash with unknown reference type * Fix more memory issues detected by valgrind * lspst - add usage mesage and option parsing using getopt (SourceForge #304199) * Fix crash caused by invalid free calls * Fix crash when email subject is empty * Fix memory and information leak in hex debug dump LibPST 0.6.3 (2008-01-13) =============================== * More type consistency issues found by splint. LibPST 0.6.2 (2008-01-12) =============================== * More fixes for Outlook 2003 64 bit parsing. * All buffer sizes changed to size_t, all file offsets changed to off_t, all function names start with pst_, many other type consistency issues found by splint. Many changes to #llx in debug printing for 64 bit items. All id values are now uint64_t. LibPST 0.6.1 (2008-01-06) =============================== * Outlook 2003 64 bit parsing. Some documentation from Alexander Grau and patches from Sean Loaring . * fix from Antonio Palama for email items that happen to have item->contact non null, and were being processed as contacts. * Add large file support so we can read .pst files larger than 2gb. * Change lspst to be similar to readpst, properly using recursion to walk the tree, and testing item types. Add a man page for lspst. LibPST 0.5.12 (2007-10-02) =============================== * security fix from Brad Hards for buffer overruns in liv-zemple decoding for corrupted or malicious pst files. LibPST 0.5.11 (2007-08-24) =============================== * fix from Stevens Miller for unitialized variable. LibPST 0.5.10 (2007-08-20) =============================== * fix yet more valgrind errors - finally have a clean memory check. * restructure readpst.c for proper recursive tree walk. * buffer overrun test was backwards, introduced at 0.5.6 * fix broken email attachments, introduced at 0.5.6 LibPST 0.5.9 (2007-08-12) =============================== * fix more valgrind errors. LibPST 0.5.8 (2007-08-10) =============================== * fix more valgrind errors. lzfu_decompress needs to return the actual buffer size, since the lz header overestimates the size. This caused base64_encode to encode undefined bytes into the email attachment. LibPST 0.5.7 (2007-08-09) =============================== * fix valgrind errors, using uninitialized data. * improve debug logging and readpstlog for indented listings. * cleanup documentation. LibPST 0.5.6 (2007-07-15) =============================== * Fix to allow very small pst files with only one node in the tree. We were mixing signed/unsigned types in comparisons. * More progress decoding the basic structure 7c blocks. Many four byte values may be ID2 indices with data outside the buffer. * Start using doxygen to generate internal documentation. LibPST 0.5.5 (2007-07-10) =============================== * merge the following changes from Joe Nahmias version: * Lots of memory fixes. Thanks to Nigel Horne for his assistance tracking these down! * Fixed creation of vCards from contacts, thanks to Nigel Horne for his help with this! * fix for MIME multipart/alternative attachments. * added -c options to readpst manpage. * use 8.3 attachment filename if long filename isn't available. * new -b option to skip rtf-body.rtf attachments. * fix format of From header lines in mbox files. * Add more appointment fields, thanks to Chris Halls for tracking them down! LibPST 0.5.4 (2006-02-25) =============================== * patches from Arne, adding MH mode, remove leading zeros from the generated numbered filenames starting with one rather than zero. Miscellaneous code cleanup. * document the "7c" descriptor block format. LibPST 0.5.3 (2006-02-20) =============================== * switch to gnu autoconf/automake. This breaks the MS VC++ projects since the source code is now in the src subdirectory. * documentation switched to xml, building man pages and html from the master xml copy. * include rpm .spec file for building src and binary rpms. LibPST 0.5.2 (2006-02-18) =============================== * Added pst2ldif to convert the contacts to ldif format for import into ldap databases. * Major changes to libpst.c to properly use the node depth values from the b-tree nodes. We also use the item count values in the nodes rather than trying to guess how many items are active. * Cleanup whitespace - using tabs for every four columns. LibPST 0.5.1 (17 November 2004) =============================== Well, alot has happened since the last release of libpst. Release / Management: * The project has forked! The new maintainer is Joseph Nahmias. * We have changed hosting sites, thanks to sourceforge for hosting to this point. From this point forward we will be using alioth.debian.org. * The project is now using SubVersioN for source control. You can get the latest code by running: svn co svn://svn.debian.org/svn/libpst/trunk . * See for more information. Code Changes: * Added lspst program to list items in a PST. Still incomplete. * Added vim folding markers to readpst.c * avoid the pseudo-prologue that MS prepends to the email headers * fix build on msvc, since it doesn't have sys/param.h * Re-vamped Makefile: * Only define CFLAGS in Makefileif missing * fixed {un,}install targets in Makefile * Fixed up build process in Makefile * Added mozilla conversion script from David Binard * Fixed bogus creation of readpst.log on every invocation * escaped dashes and apostrophe in manpages * Updated TODO * added manpages from debian pkg * fix escaped-string length count to consider '\n', thanks to Paul Bakker . * ensure there's a blank line between header and body patch from (SourceForge #890745). * Apply accumulated endian-related patches * Removed unused files, upstream's debian/ dir -- Joe Nahmias LibPST v0.5 =========== It is with GREAT relief that I bring you version 0.5 of the LibPST tools! Through great difficulties, this tool has survived and expanded to become even better. The changes are as follows: * RTF support. We can now decompress RTF bodies in emails, and are saved as attachments * Better support in reading the indexes. Fixed many bugs with them * Improved reliability. "Now we are getting somewhere!" * Improved compiling. Hopefully we won't be hitting too many compile errors now. * vCard handling. Contacts are now exported as vCard entries. * vEvent handling. Support has begun on exporting Calendar entries as events * Support for Journal entries has also begun If you have any problems with this release, don't hesitate to contact me. These changes come to you, as always, free under the GPL license!! What a wonderful thing it is. It does mean that you can write your own program off of this library and distribute it also for free. However, anyone with commercial interests for developing applications they will be charging for are encouraged to get in touch with me, as I am sure we can come to some arrangement. Dave Smith LibPST v0.4.3 ============= Bug fix release. No extra functionality Dave Smith LibPST v0.4.2 ============= The debug system has had an overhaul. The debug messages are no longer printed to the screen when they are enabled. They are dumped to a binary file. There is another utility called "readlog" that I have written to handle these log files. It should make it easier to selectively view bits of a log file. It also shows the position that the log message was printed from. There is a new switch in readpst. It is -d. It enables the user to specify the log file which the binary log is written to. If the switch isn't used, the default file of "readpst.log" is used. The code is now Visual C++ compatible. It has compiled on Visual C++ .net Standard edition, and produces the readpst.exe file. Use the project file included in this distribution. There have been minor improvements elsewhere too. LibPST v0.4.1 ============= Fixed a couple more bugs. Is it me or do bugs just insert themselves in random, hard to find places! Cured a few problems with regard to emails with multiple embeded items. They are not fully re-created using Mime-types, but are accessible with the -S switch (which saves everything as seperate items) Fixed a problem reading the first index. Back sliders are now detected. (ie when the value following the current one is smaller, not bigger!) Added some error messages when we try and read outside of the PST file, this was causing a few problems before, cause the return value wasn't always checked, so it was possible to be reading random data, and trying to make sense of it! Anyway, if you find any problems, don't hesitate to mail me Dave Smith LibPST v0.4 =========== Fixed a nasty bug that occasionally corrupted attachments. Another bug with regard to reading of indexes (also occasional). Another output method has been added which is called "Seperate". It is activated with the -S switch. It operates in the following manor: |--Inbox-->000000 | 000001 | 000002 |--Sentmail-->0000000 | 0000001 | 0000002 All the emails are stored in seperate files counting from 0 upwards, in a folder named as the PST folder. When an email has an attachment, it is saved as a seperate file. The filename for the attachment is made up of 2 parts, the first is the email number to which it belongs, the second is its filename. The should now be runnable on big-endian machines, if the define.h file is first modified. The #define LITTLE_ENDIAN must be commented out, and the #define BIG_ENDIAN must be uncommented. More verbose error messages have been added. Apparently people got confused when the program stopped for no visible reason. This has now been resolved. Thanks for the continued support of all people involved. Dave Smith Libpst v0.3.4 ============= Several more fixes. An Infinite loop and incorrect interpreting of item index attributes. Work has started on making the code executable on big endian CPUs. At present it should work with Linux on these CPUs, but I would appreciate it if you could provide feedback with regard to it's performance. I am also working with some other people at make it operate on Solaris. A whole load more items are now recognized by the Item records. With more items in Emails and Folders. I haven't got to the Contacts yet. Anyway, this is what I would call a minor feature enhancment and bugfix release. Dave Smith LibPST v0.3.3 ============= Fixed several items. Mainly memory leaks. Loads of them! oops.. I have added a new program, mainly of debugging, which when passed an ID value and a pst file, will extract and decrypt that ID from the pst file. I don't see it being a huge attraction, or of much use to most people, but it is another example of writing an application to use the libpst interface. Another fix was in the reading of the item index. This has hopefully now been corrected. The result of this bug was that not all the emails in a folder were converted. Hopefully you should have more luck now. Dave Smith LibPST v0.3.2 ============= Quick bugfix release. There was a bug in the decryption of the basic encryption that outlook uses. One byte, 0x6c, was incorrectly decrypted to 0x6c instead of 0xcd. This release fixes this bug. Sorry... LibPST v0.3.1 ============= Minor improvements. Fixed bug when linking multiple blocks together, so now the linking blocks are not "encrypted" when trying to read them. LibPST v0.3 =========== A lot of bug fixing has been done for this release. Testing has been done on the creation of the files by readpst. Better handling of large binaries being extracted from the PST file has been implemented. Quite a few reports have come in about not being able to compile on Darwin. This could be down to using macros with variable parameter lists. This has now been changed to use C functions with variable parameters. I hope this fixes a lot of problems. Added support for recreating the folder structure into normal directories. For Instance: Personal Folders |-Inbox | |-Jokes | |-Meetings |-Send Items each folder containing an mbox file with the correct emails for that folder. Dave Smith LibPST v0.3 beta1 ================= Again, a shed load of enhancements. More work has been done on the mime creation. A bug has been fixed that was letting part of the attachments that were created disappear. A major enhancement is that "compressible encryption" support has been added. This was an incredibly simple method to use. It is basically a ceasar cipher. It has been noted by several users already that the PST password that Outlook uses, serves *no purpose*. It is not used to encrypt the PST, it is mearly stored there. This means that the readpst application is able to convert PST files without knowing the password. Microsoft have some explaning to do! Output files are now not overwritten if they already exist. This means that if you have two folders in your PST file named "fred", the first one encountered will be named "fred" and the second one will be named "fred00000001". As you can see, there is enough room there for many duplicate names! Output filenames are now restricted. Any "/" or "\" characters in the name are replaced with "_". If you find that there are any other characters that need to be changed, could you please make me aware! Thanks to Berry Wizard for help with supporting the encryption. Thanks to Auke Kok, Carolus Walraven and Yogesh Kumar Guatam for providing debugging information and testing. Dave Smith LibPST v0.2 beta1 ================= Hello once more... Attachments are now re-created in mime format. The method is very crude and could be prone to over generalisation. Please test this version, and if attachments are not recreated correctly, please send me the email (complete message source) of the original and converted. Cheers. I hope this will work for everyone who uses this program, but reality can be very different! Let us see how it goes... Dave Smith LibPST v0.2 alpha1 =========== Hello! Some improvements. The internal code has been changed so that attachments are now processed and loaded into the structures. The readpst program is not finished yet. It needs to convert these binary structs into mime data. At present it just saves them to the current directory, overwriting any previous files with the attachment name. Improvements over previous version: * KMail output is supported - if the "-k" flag is specified, all the directory hierarchy is created using the KMail standard * Lots of bugs and memory leaks fixed Usage: ReadPST v0.2alpha1 implementing LibPST v0.2alpha1 Usage: ./readpst [OPTIONS] {PST FILENAME} OPTIONS: -h - Help. This screen -k - KMail. Output in kmail format -o - Output Dir. Directory to write files to. CWD is changed *after* opening pst file -V - Version. Display program version If you want to view lots of debug output, modify a line in "define.h" from "//#define DEBUG_ALL" to "#define DEBUG_ALL". It would then be advisable to pipe all output to a log file: ./readpst -o out pst_file &> logfile Dave Smith LibPST v0.1 =========== Hi Folks! This has been a long, hard slog, but I now feel that I have got somewhere useful. The included program "main" is able to read an Outlook PST file and dump the emails into mbox files, separating each folder into a different mbox file. All the mbox files are stored in the current directory and no attempt is yet made to organise these files into a directory hierarchy. This would not be too difficult to achieve though. Email attachments are not yet handled, neither are Contacts. There is no pretty interface yet, but you can convert a PST file in the following manner ./main {path to PST file} This is very much a work in progress, but I thought I should release this code so that people can lose their conception that outlook files will never be converted to Linux. I am intending that the code I am writing will be developed into greater applications to provide USEFUL tools for accessing and converting PST files into a variety of formats. One point I feel I should make is that Outlook, by default, creates "Compressible Encryption" PST files. I have not, as yet, attempted to write any decryption routines, so you will not be able to convert these files. However, if you create a new PST file and choose not to make an encrypted one, you can copy all your emails into this new one and then convert the unencrypted one. I hope you enjoy, Dave Smith diff --git a/src/readpst.c b/src/readpst.c index c0d1858..958ad9b 100644 --- a/src/readpst.c +++ b/src/readpst.c @@ -1,1661 +1,1631 @@ /*** * readpst.c * Part of the LibPST project * Written by David Smith * dave.s@earthcorp.com */ #include "define.h" #include "lzfu.h" #define OUTPUT_TEMPLATE "%s" #define OUTPUT_KMAIL_DIR_TEMPLATE ".%s.directory" #define KMAIL_INDEX ".%s.index" #define SEP_MAIL_FILE_TEMPLATE "%i" // max size of the c_time char*. It will store the date of the email #define C_TIME_SIZE 500 struct file_ll { char *name; char *dname; FILE * output; int32_t stored_count; int32_t item_count; int32_t skip_count; int32_t type; }; void process(pst_item *outeritem, pst_desc_tree *d_ptr); void write_email_body(FILE *f, char *body); void removeCR(char *c); void usage(); void version(); char* mk_kmail_dir(char* fname); int close_kmail_dir(); char* mk_recurse_dir(char* dir, int32_t folder_type); int close_recurse_dir(); char* mk_separate_dir(char *dir); int close_separate_dir(); int mk_separate_file(struct file_ll *f); char* my_stristr(char *haystack, char *needle); void check_filename(char *fname); void write_separate_attachment(char f_name[], pst_item_attach* attach, int attach_num, pst_file* pst); void write_embedded_message(FILE* f_output, pst_item_attach* attach, char *boundary, pst_file* pf, char** extra_mime_headers); void write_inline_attachment(FILE* f_output, pst_item_attach* attach, char *boundary, pst_file* pst); void header_has_field(char *header, char *field, int *flag); void header_get_subfield(char *field, const char *subfield, char *body_subfield, size_t size_subfield); char* header_get_field(char *header, char *field); char* header_end_field(char *field); void header_strip_field(char *header, char *field); int test_base64(char *body); void find_html_charset(char *html, char *charset, size_t charsetlen); void find_rfc822_headers(char** extra_mime_headers); void write_body_part(FILE* f_output, pst_string *body, char *mime, char *charset, char *boundary, pst_file* pst); void write_normal_email(FILE* f_output, char f_name[], pst_item* item, int mode, int mode_MH, pst_file* pst, int save_rtf, char** extra_mime_headers); void write_vcard(FILE* f_output, pst_item *item, pst_item_contact* contact, char comment[]); void write_appointment(FILE* f_output, pst_item *item, pst_item_appointment* appointment, FILETIME* create_date, FILETIME* modify_date); void create_enter_dir(struct file_ll* f, pst_item *item); void close_enter_dir(struct file_ll *f); const char* prog_name; char* output_dir = "."; char* kmail_chdir = NULL; // Normal mode just creates mbox format files in the current directory. Each file is named // the same as the folder's name that it represents #define MODE_NORMAL 0 // KMail mode creates a directory structure suitable for being used directly // by the KMail application #define MODE_KMAIL 1 // recurse mode creates a directory structure like the PST file. Each directory // contains only one file which stores the emails in mbox format. #define MODE_RECURSE 2 // separate mode creates the same directory structure as recurse. The emails are stored in // separate files, numbering from 1 upward. Attachments belonging to the emails are // saved as email_no-filename (e.g. 1-samplefile.doc or 000001-Attachment2.zip) #define MODE_SEPARATE 3 -// Decrypt the whole file (even the parts that aren't encrypted) and ralph it to stdout -#define MODE_DECSPEW 4 - // Output Normal just prints the standard information about what is going on #define OUTPUT_NORMAL 0 // Output Quiet is provided so that only errors are printed #define OUTPUT_QUIET 1 // default mime-type for attachments that have a null mime-type #define MIME_TYPE_DEFAULT "application/octet-stream" #define RFC822 "message/rfc822" // output mode for contacts #define CMODE_VCARD 0 #define CMODE_LIST 1 // output mode for deleted items #define DMODE_EXCLUDE 0 #define DMODE_INCLUDE 1 // output settings for RTF bodies // filename for the attachment #define RTF_ATTACH_NAME "rtf-body.rtf" // mime type for the attachment #define RTF_ATTACH_TYPE "application/rtf" // global settings int mode = MODE_NORMAL; int mode_MH = 0; // a submode of MODE_SEPARATE int output_mode = OUTPUT_NORMAL; int contact_mode = CMODE_VCARD; int deleted_mode = DMODE_EXCLUDE; int contact_mode_specified = 0; int overwrite = 0; int save_rtf_body = 1; pst_file pstfile; regex_t meta_charset_pattern; void process(pst_item *outeritem, pst_desc_tree *d_ptr) { struct file_ll ff; pst_item *item = NULL; DEBUG_ENT("process"); memset(&ff, 0, sizeof(ff)); create_enter_dir(&ff, outeritem); for (; d_ptr; d_ptr = d_ptr->next) { DEBUG_MAIN(("main: New item record\n")); if (!d_ptr->desc) { ff.skip_count++; DEBUG_WARN(("main: ERROR item's desc record is NULL\n")); continue; } DEBUG_MAIN(("main: Desc Email ID %#"PRIx64" [d_ptr->d_id = %#"PRIx64"]\n", d_ptr->desc->i_id, d_ptr->d_id)); item = pst_parse_item(&pstfile, d_ptr, NULL); DEBUG_MAIN(("main: About to process item\n")); if (!item) { ff.skip_count++; DEBUG_MAIN(("main: A NULL item was seen\n")); continue; } if (item->subject.str) { DEBUG_EMAIL(("item->subject = %s\n", item->subject.str)); } if (item->folder && item->file_as.str) { DEBUG_MAIN(("Processing Folder \"%s\"\n", item->file_as.str)); if (output_mode != OUTPUT_QUIET) printf("Processing Folder \"%s\"\n", item->file_as.str); ff.item_count++; if (d_ptr->child && (deleted_mode == DMODE_INCLUDE || strcasecmp(item->file_as.str, "Deleted Items"))) { //if this is a non-empty folder other than deleted items, we want to recurse into it process(item, d_ptr->child); } } else if (item->contact && (item->type == PST_TYPE_CONTACT)) { if (!ff.type) ff.type = item->type; DEBUG_MAIN(("main: Processing Contact\n")); if (ff.type != PST_TYPE_CONTACT) { ff.skip_count++; DEBUG_MAIN(("main: I have a contact, but the folder type %"PRIi32" isn't a contacts folder. Skipping it\n", ff.type)); } else { ff.item_count++; if (mode == MODE_SEPARATE) mk_separate_file(&ff); if (contact_mode == CMODE_VCARD) { pst_convert_utf8_null(item, &item->comment); write_vcard(ff.output, item, item->contact, item->comment.str); } else { pst_convert_utf8(item, &item->contact->fullname); pst_convert_utf8(item, &item->contact->address1); fprintf(ff.output, "%s <%s>\n", item->contact->fullname.str, item->contact->address1.str); } } } else if (item->email && (item->type == PST_TYPE_NOTE || item->type == PST_TYPE_REPORT)) { if (!ff.type) ff.type = item->type; DEBUG_MAIN(("main: Processing Email\n")); if ((ff.type != PST_TYPE_NOTE) && (ff.type != PST_TYPE_REPORT)) { ff.skip_count++; DEBUG_MAIN(("main: I have an email type %"PRIi32", but the folder type %"PRIi32" isn't an email folder. Skipping it\n", item->type, ff.type)); } else { char *extra_mime_headers = NULL; ff.item_count++; if (mode == MODE_SEPARATE) mk_separate_file(&ff); write_normal_email(ff.output, ff.name, item, mode, mode_MH, &pstfile, save_rtf_body, &extra_mime_headers); } } else if (item->journal && (item->type == PST_TYPE_JOURNAL)) { if (!ff.type) ff.type = item->type; DEBUG_MAIN(("main: Processing Journal Entry\n")); if (ff.type != PST_TYPE_JOURNAL) { ff.skip_count++; DEBUG_MAIN(("main: I have a journal entry, but the folder type %"PRIi32" isn't a journal folder. Skipping it\n", ff.type)); } else { ff.item_count++; if (mode == MODE_SEPARATE) mk_separate_file(&ff); fprintf(ff.output, "BEGIN:VJOURNAL\n"); if (item->subject.str) { pst_convert_utf8(item, &item->subject); fprintf(ff.output, "SUMMARY:%s\n", pst_rfc2426_escape(item->subject.str)); } if (item->body.str) { pst_convert_utf8(item, &item->body); fprintf(ff.output, "DESCRIPTION:%s\n", pst_rfc2426_escape(item->body.str)); } if (item->journal->start) fprintf(ff.output, "DTSTART;VALUE=DATE-TIME:%s\n", pst_rfc2445_datetime_format(item->journal->start)); fprintf(ff.output, "END:VJOURNAL\n\n"); } } else if (item->appointment && (item->type == PST_TYPE_APPOINTMENT)) { if (!ff.type) ff.type = item->type; DEBUG_MAIN(("main: Processing Appointment Entry\n")); if (ff.type != PST_TYPE_APPOINTMENT) { ff.skip_count++; DEBUG_MAIN(("main: I have an appointment, but the folder type %"PRIi32" isn't an appointment folder. Skipping it\n", ff.type)); } else { ff.item_count++; if (mode == MODE_SEPARATE) mk_separate_file(&ff); write_appointment(ff.output, item, item->appointment, item->create_date, item->modify_date); } } else if (item->message_store) { // there should only be one message_store, and we have already done it ff.skip_count++; DEBUG_MAIN(("item with message store content, type %i %s folder type %i, skipping it\n", item->type, item->ascii_type, ff.type)); } else { ff.skip_count++; DEBUG_MAIN(("main: Unknown item type %i (%s) name (%s)\n", item->type, item->ascii_type, item->file_as.str)); } pst_freeItem(item); } close_enter_dir(&ff); DEBUG_RET(); } int main(int argc, char* const* argv) { pst_item *item = NULL; pst_desc_tree *d_ptr; char * fname = NULL; char *d_log = NULL; int c,x; char *temp = NULL; //temporary char pointer prog_name = argv[0]; time_t now = time(NULL); srand((unsigned)now); if (regcomp(&meta_charset_pattern, "]*content=\"[^>]*charset=([^>\";]*)[\";]", REG_ICASE | REG_EXTENDED)) { printf("cannot compile regex pattern to find content charset in html bodies\n"); exit(3); } // command-line option handling - while ((c = getopt(argc, argv, "bCc:Dd:hko:qrSMVw"))!= -1) { + while ((c = getopt(argc, argv, "bc:Dd:hko:qrSMVw"))!= -1) { switch (c) { case 'b': save_rtf_body = 0; break; - case 'C': - mode = MODE_DECSPEW; - break; case 'c': if (optarg && optarg[0]=='v') { contact_mode=CMODE_VCARD; contact_mode_specified = 1; } else if (optarg && optarg[0]=='l') { contact_mode=CMODE_LIST; contact_mode_specified = 1; } else { usage(); exit(0); } break; case 'D': deleted_mode = DMODE_INCLUDE; break; case 'd': d_log = optarg; break; case 'h': usage(); exit(0); break; case 'V': version(); exit(0); break; case 'k': mode = MODE_KMAIL; break; case 'M': mode = MODE_SEPARATE; mode_MH = 1; break; case 'o': output_dir = optarg; break; case 'q': output_mode = OUTPUT_QUIET; break; case 'r': mode = MODE_RECURSE; break; case 'S': mode = MODE_SEPARATE; mode_MH = 0; break; case 'w': overwrite = 1; break; default: usage(); exit(1); break; } } if (argc > optind) { fname = argv[optind]; } else { usage(); exit(2); } #ifdef DEBUG_ALL // force a log file if (!d_log) d_log = "readpst.log"; #endif // defined DEBUG_ALL DEBUG_INIT(d_log); DEBUG_REGISTER_CLOSE(); DEBUG_ENT("main"); - if (mode == MODE_DECSPEW) { - FILE *fp; - char buf[1024]; - size_t l = 0; - if (NULL == (fp = fopen(fname, "rb"))) { - WARN(("Couldn't open file %s\n", fname)); - DEBUG_RET(); - return 1; - } - - while (0 != (l = fread(buf, 1, 1024, fp))) { - if (0 != pst_decrypt(0, buf, l, PST_COMP_ENCRYPT)) - WARN(("pst_decrypt() failed (I'll try to continue)\n")); - - if (l != pst_fwrite(buf, 1, l, stdout)) { - WARN(("Couldn't output to stdout?\n")); - DEBUG_RET(); - return 1; - } - } - DEBUG_RET(); - return 0; - } - if (output_mode != OUTPUT_QUIET) printf("Opening PST file and indexes...\n"); RET_DERROR(pst_open(&pstfile, fname), 1, ("Error opening File\n")); RET_DERROR(pst_load_index(&pstfile), 2, ("Index Error\n")); pst_load_extended_attributes(&pstfile); if (chdir(output_dir)) { x = errno; pst_close(&pstfile); DEBUG_RET(); DIE(("main: Cannot change to output dir %s: %s\n", output_dir, strerror(x))); } if (output_mode != OUTPUT_QUIET) printf("About to start processing first record...\n"); d_ptr = pstfile.d_head; // first record is main record item = pst_parse_item(&pstfile, d_ptr, NULL); if (!item || !item->message_store) { DEBUG_RET(); DIE(("main: Could not get root record\n")); } // default the file_as to the same as the main filename if it doesn't exist if (!item->file_as.str) { if (!(temp = strrchr(fname, '/'))) if (!(temp = strrchr(fname, '\\'))) temp = fname; else temp++; // get past the "\\" else temp++; // get past the "/" item->file_as.str = (char*)pst_malloc(strlen(temp)+1); strcpy(item->file_as.str, temp); item->file_as.is_utf8 = 1; DEBUG_MAIN(("file_as was blank, so am using %s\n", item->file_as.str)); } DEBUG_MAIN(("main: Root Folder Name: %s\n", item->file_as.str)); d_ptr = pst_getTopOfFolders(&pstfile, item); if (!d_ptr) { DEBUG_RET(); DIE(("Top of folders record not found. Cannot continue\n")); } process(item, d_ptr->child); // do the children of TOPF pst_freeItem(item); pst_close(&pstfile); DEBUG_RET(); regfree(&meta_charset_pattern); return 0; } void write_email_body(FILE *f, char *body) { char *n = body; DEBUG_ENT("write_email_body"); DEBUG_INFO(("buffer pointer %p\n", body)); while (n) { if (strncmp(body, "From ", 5) == 0) fprintf(f, ">"); if ((n = strchr(body, '\n'))) { n++; pst_fwrite(body, n-body, 1, f); //write just a line body = n; } } pst_fwrite(body, strlen(body), 1, f); DEBUG_RET(); } void removeCR (char *c) { // converts \r\n to \n char *a, *b; DEBUG_ENT("removeCR"); a = b = c; while (*a != '\0') { *b = *a; if (*a != '\r') b++; a++; } *b = '\0'; DEBUG_RET(); } void usage() { DEBUG_ENT("usage"); version(); printf("Usage: %s [OPTIONS] {PST FILENAME}\n", prog_name); printf("OPTIONS:\n"); printf("\t-V\t- Version. Display program version\n"); printf("\t-C\t- Decrypt (compressible encryption) the entire file and output on stdout (not typically useful)\n"); printf("\t-D\t- Include deleted items in output\n"); printf("\t-M\t- MH. Write emails in the MH format\n"); printf("\t-S\t- Separate. Write emails in the separate format\n"); printf("\t-b\t- Don't save RTF-Body attachments\n"); printf("\t-c[v|l]\t- Set the Contact output mode. -cv = VCard, -cl = EMail list\n"); printf("\t-d \t- Debug to file. This is a binary log. Use readpstlog to print it\n"); printf("\t-h\t- Help. This screen\n"); printf("\t-k\t- KMail. Output in kmail format\n"); printf("\t-o \t- Output directory to write files to. CWD is changed *after* opening pst file\n"); printf("\t-q\t- Quiet. Only print error messages\n"); printf("\t-r\t- Recursive. Output in a recursive format\n"); printf("\t-w\t- Overwrite any output mbox files\n"); DEBUG_RET(); } void version() { DEBUG_ENT("version"); printf("ReadPST / LibPST v%s\n", VERSION); #if BYTE_ORDER == BIG_ENDIAN printf("Big Endian implementation being used.\n"); #elif BYTE_ORDER == LITTLE_ENDIAN printf("Little Endian implementation being used.\n"); #else # error "Byte order not supported by this library" #endif #ifdef __GNUC__ printf("GCC %d.%d : %s %s\n", __GNUC__, __GNUC_MINOR__, __DATE__, __TIME__); #endif DEBUG_RET(); } char *mk_kmail_dir(char *fname) { //change to that directory //make a directory based on OUTPUT_KMAIL_DIR_TEMPLATE //allocate space for OUTPUT_TEMPLATE and form a char* with fname //return that value char *dir, *out_name, *index; int x; DEBUG_ENT("mk_kmail_dir"); if (kmail_chdir && chdir(kmail_chdir)) { x = errno; DIE(("mk_kmail_dir: Cannot change to directory %s: %s\n", kmail_chdir, strerror(x))); } dir = malloc(strlen(fname)+strlen(OUTPUT_KMAIL_DIR_TEMPLATE)+1); sprintf(dir, OUTPUT_KMAIL_DIR_TEMPLATE, fname); check_filename(dir); if (D_MKDIR(dir)) { //error occured if (errno != EEXIST) { x = errno; DIE(("mk_kmail_dir: Cannot create directory %s: %s\n", dir, strerror(x))); } } kmail_chdir = realloc(kmail_chdir, strlen(dir)+1); strcpy(kmail_chdir, dir); free (dir); //we should remove any existing indexes created by KMail, cause they might be different now index = malloc(strlen(fname)+strlen(KMAIL_INDEX)+1); sprintf(index, KMAIL_INDEX, fname); unlink(index); free(index); out_name = malloc(strlen(fname)+strlen(OUTPUT_TEMPLATE)+1); sprintf(out_name, OUTPUT_TEMPLATE, fname); DEBUG_RET(); return out_name; } int close_kmail_dir() { // change .. int x; DEBUG_ENT("close_kmail_dir"); if (kmail_chdir) { //only free kmail_chdir if not NULL. do not change directory free(kmail_chdir); kmail_chdir = NULL; } else { if (chdir("..")) { x = errno; DIE(("close_kmail_dir: Cannot move up dir (..): %s\n", strerror(x))); } } DEBUG_RET(); return 0; } // this will create a directory by that name, then make an mbox file inside // that dir. any subsequent dirs will be created by name, and they will // contain mbox files char *mk_recurse_dir(char *dir, int32_t folder_type) { int x; char *out_name; DEBUG_ENT("mk_recurse_dir"); check_filename(dir); if (D_MKDIR (dir)) { if (errno != EEXIST) { // not an error because it exists x = errno; DIE(("mk_recurse_dir: Cannot create directory %s: %s\n", dir, strerror(x))); } } if (chdir (dir)) { x = errno; DIE(("mk_recurse_dir: Cannot change to directory %s: %s\n", dir, strerror(x))); } switch (folder_type) { case PST_TYPE_APPOINTMENT: out_name = strdup("calendar"); break; case PST_TYPE_CONTACT: out_name = strdup("contacts"); break; case PST_TYPE_JOURNAL: out_name = strdup("journal"); break; case PST_TYPE_STICKYNOTE: case PST_TYPE_TASK: case PST_TYPE_NOTE: case PST_TYPE_OTHER: case PST_TYPE_REPORT: default: out_name = strdup("mbox"); break; } DEBUG_RET(); return out_name; } int close_recurse_dir() { int x; DEBUG_ENT("close_recurse_dir"); if (chdir("..")) { x = errno; DIE(("close_recurse_dir: Cannot go up dir (..): %s\n", strerror(x))); } DEBUG_RET(); return 0; } char *mk_separate_dir(char *dir) { size_t dirsize = strlen(dir) + 10; char dir_name[dirsize]; int x = 0, y = 0; DEBUG_ENT("mk_separate_dir"); do { if (y == 0) snprintf(dir_name, dirsize, "%s", dir); else snprintf(dir_name, dirsize, "%s" SEP_MAIL_FILE_TEMPLATE, dir, y); // enough for 9 digits allocated above check_filename(dir_name); DEBUG_MAIN(("about to try creating %s\n", dir_name)); if (D_MKDIR(dir_name)) { if (errno != EEXIST) { // if there is an error, and it doesn't already exist x = errno; DIE(("mk_separate_dir: Cannot create directory %s: %s\n", dir, strerror(x))); } } else { break; } y++; } while (overwrite == 0); if (chdir(dir_name)) { x = errno; DIE(("mk_separate_dir: Cannot change to directory %s: %s\n", dir, strerror(x))); } if (overwrite) { // we should probably delete all files from this directory #if !defined(WIN32) && !defined(__CYGWIN__) DIR * sdir = NULL; struct dirent *dirent = NULL; struct stat filestat; if (!(sdir = opendir("./"))) { WARN(("mk_separate_dir: Cannot open dir \"%s\" for deletion of old contents\n", "./")); } else { while ((dirent = readdir(sdir))) { if (lstat(dirent->d_name, &filestat) != -1) if (S_ISREG(filestat.st_mode)) { if (unlink(dirent->d_name)) { y = errno; DIE(("mk_separate_dir: unlink returned error on file %s: %s\n", dirent->d_name, strerror(y))); } } } } #endif } // we don't return a filename here cause it isn't necessary. DEBUG_RET(); return NULL; } int close_separate_dir() { int x; DEBUG_ENT("close_separate_dir"); if (chdir("..")) { x = errno; DIE(("close_separate_dir: Cannot go up dir (..): %s\n", strerror(x))); } DEBUG_RET(); return 0; } int mk_separate_file(struct file_ll *f) { const int name_offset = 1; DEBUG_ENT("mk_separate_file"); DEBUG_MAIN(("opening next file to save email\n")); if (f->item_count > 999999999) { // bigger than nine 9's DIE(("mk_separate_file: The number of emails in this folder has become too high to handle")); } sprintf(f->name, SEP_MAIL_FILE_TEMPLATE, f->item_count + name_offset); if (f->output) fclose(f->output); f->output = NULL; check_filename(f->name); if (!(f->output = fopen(f->name, "w"))) { DIE(("mk_separate_file: Cannot open file to save email \"%s\"\n", f->name)); } DEBUG_RET(); return 0; } char *my_stristr(char *haystack, char *needle) { // my_stristr varies from strstr in that its searches are case-insensitive char *x=haystack, *y=needle, *z = NULL; if (!haystack || !needle) { return NULL; } while (*y != '\0' && *x != '\0') { if (tolower(*y) == tolower(*x)) { // move y on one y++; if (!z) { z = x; // store first position in haystack where a match is made } } else { y = needle; // reset y to the beginning of the needle z = NULL; // reset the haystack storage point } x++; // advance the search in the haystack } // If the haystack ended before our search finished, it's not a match. if (*y != '\0') return NULL; return z; } void check_filename(char *fname) { char *t = fname; DEBUG_ENT("check_filename"); if (!t) { DEBUG_RET(); return; } while ((t = strpbrk(t, "/\\:"))) { // while there are characters in the second string that we don't want *t = '_'; //replace them with an underscore } DEBUG_RET(); } void write_separate_attachment(char f_name[], pst_item_attach* attach, int attach_num, pst_file* pst) { FILE *fp = NULL; int x = 0; char *temp = NULL; // If there is a long filename (filename2) use that, otherwise // use the 8.3 filename (filename1) char *attach_filename = (attach->filename2.str) ? attach->filename2.str : attach->filename1.str; DEBUG_ENT("write_separate_attachment"); if (!attach->data.data) { // make sure we can fetch data from the id pst_index_ll *ptr = pst_getID(pst, attach->i_id); if (!ptr) { DEBUG_WARN(("Couldn't find i_id %#"PRIx64". Cannot save attachment to file\n", attach->i_id)); DEBUG_RET(); return; } } check_filename(f_name); if (!attach_filename) { // generate our own (dummy) filename for the attachement temp = pst_malloc(strlen(f_name)+15); sprintf(temp, "%s-attach%i", f_name, attach_num); } else { // have an attachment name, make sure it's unique temp = pst_malloc(strlen(f_name)+strlen(attach_filename)+15); do { if (fp) fclose(fp); if (x == 0) sprintf(temp, "%s-%s", f_name, attach_filename); else sprintf(temp, "%s-%s-%i", f_name, attach_filename, x); } while ((fp = fopen(temp, "r")) && ++x < 99999999); if (x > 99999999) { DIE(("error finding attachment name. exhausted possibilities to %s\n", temp)); } } DEBUG_EMAIL(("Saving attachment to %s\n", temp)); if (!(fp = fopen(temp, "w"))) { WARN(("write_separate_attachment: Cannot open attachment save file \"%s\"\n", temp)); } else { (void)pst_attach_to_file(pst, attach, fp); fclose(fp); } if (temp) free(temp); DEBUG_RET(); } void write_embedded_message(FILE* f_output, pst_item_attach* attach, char *boundary, pst_file* pf, char** extra_mime_headers) { pst_index_ll *ptr; DEBUG_ENT("write_embedded_message"); fprintf(f_output, "\n--%s\n", boundary); fprintf(f_output, "Content-Type: %s\n\n", attach->mimetype.str); ptr = pst_getID(pf, attach->i_id); pst_desc_tree d_ptr; d_ptr.d_id = 0; d_ptr.parent_d_id = 0; d_ptr.assoc_tree = NULL; d_ptr.desc = ptr; d_ptr.no_child = 0; d_ptr.prev = NULL; d_ptr.next = NULL; d_ptr.parent = NULL; d_ptr.child = NULL; d_ptr.child_tail = NULL; pst_item *item = pst_parse_item(pf, &d_ptr, attach->id2_head); write_normal_email(f_output, "", item, MODE_NORMAL, 0, pf, 0, extra_mime_headers); pst_freeItem(item); DEBUG_RET(); } void write_inline_attachment(FILE* f_output, pst_item_attach* attach, char *boundary, pst_file* pst) { char *attach_filename; DEBUG_ENT("write_inline_attachment"); DEBUG_EMAIL(("Attachment Size is %"PRIu64", id %#"PRIx64"\n", (uint64_t)attach->data.size, attach->i_id)); if (!attach->data.data) { // make sure we can fetch data from the id pst_index_ll *ptr = pst_getID(pst, attach->i_id); if (!ptr) { DEBUG_WARN(("Couldn't find ID pointer. Cannot save attachment to file\n")); DEBUG_RET(); return; } } fprintf(f_output, "\n--%s\n", boundary); if (!attach->mimetype.str) { fprintf(f_output, "Content-Type: %s\n", MIME_TYPE_DEFAULT); } else { fprintf(f_output, "Content-Type: %s\n", attach->mimetype.str); } fprintf(f_output, "Content-Transfer-Encoding: base64\n"); // If there is a long filename (filename2) use that, otherwise // use the 8.3 filename (filename1) attach_filename = (attach->filename2.str) ? attach->filename2.str : attach->filename1.str; if (!attach_filename) { fprintf(f_output, "Content-Disposition: inline\n\n"); } else { fprintf(f_output, "Content-Disposition: attachment; filename=\"%s\"\n\n", attach_filename); } (void)pst_attach_to_file_base64(pst, attach, f_output); fprintf(f_output, "\n\n"); DEBUG_RET(); } void header_has_field(char *header, char *field, int *flag) { DEBUG_ENT("header_has_field"); if (my_stristr(header, field) || (strncasecmp(header, field+1, strlen(field)-1) == 0)) { DEBUG_EMAIL(("header block has %s header\n", field+1)); *flag = 1; } DEBUG_RET(); } void header_get_subfield(char *field, const char *subfield, char *body_subfield, size_t size_subfield) { if (!field) return; DEBUG_ENT("header_get_subfield"); char search[60]; snprintf(search, sizeof(search), " %s=", subfield); field++; char *n = header_end_field(field); char *s = my_stristr(field, search); if (n && s && (s < n)) { char *e, *f, save; s += strlen(search); // skip over subfield= if (*s == '"') { s++; e = strchr(s, '"'); } else { e = strchr(s, ';'); f = strchr(s, '\n'); if (e && f && (f < e)) e = f; } if (!e || (e > n)) e = n; // use the trailing lf as terminator if nothing better save = *e; *e = '\0'; snprintf(body_subfield, size_subfield, "%s", s); // copy the subfield to our buffer *e = save; DEBUG_EMAIL(("body %s %s from headers\n", subfield, body_subfield)); } DEBUG_RET(); } char* header_get_field(char *header, char *field) { char *t = my_stristr(header, field); if (!t && (strncasecmp(header, field+1, strlen(field)-1) == 0)) t = header; return t; } // return pointer to \n at the end of this header field, // or NULL if this field goes to the end of the string. char *header_end_field(char *field) { char *e = strchr(field+1, '\n'); while (e && ((e[1] == ' ') || (e[1] == '\t'))) { e = strchr(e+1, '\n'); } return e; } void header_strip_field(char *header, char *field) { char *t = header_get_field(header, field); if (t) { char *e = header_end_field(t); if (e) { if (t == header) e++; // if *t is not \n, we don't want to keep the \n at *e either. while (*e != '\0') { *t = *e; t++; e++; } *t = '\0'; } else { // this was the last header field, truncate the headers *t = '\0'; } } } int test_base64(char *body) { int b64 = 0; uint8_t *b = (uint8_t *)body; DEBUG_ENT("test_base64"); while (*b != 0) { if ((*b < 32) && (*b != 9) && (*b != 10)) { DEBUG_EMAIL(("found base64 byte %d\n", (int)*b)); DEBUG_HEXDUMPC(body, strlen(body), 0x10); b64 = 1; break; } b++; } DEBUG_RET(); return b64; } void find_html_charset(char *html, char *charset, size_t charsetlen) { const int index = 1; const int nmatch = index+1; regmatch_t match[nmatch]; DEBUG_ENT("find_html_charset"); int rc = regexec(&meta_charset_pattern, html, nmatch, match, 0); if (rc == 0) { int s = match[index].rm_so; int e = match[index].rm_eo; if (s != -1) { char save = html[e]; html[e] = '\0'; snprintf(charset, charsetlen, "%s", html+s); // copy the html charset html[e] = save; DEBUG_EMAIL(("charset %s from html text\n", charset)); } else { DEBUG_EMAIL(("matching %d %d %d %d", match[0].rm_so, match[0].rm_eo, match[1].rm_so, match[1].rm_eo)); DEBUG_HEXDUMPC(html, strlen(html), 0x10); } } else { DEBUG_EMAIL(("regexec returns %d\n", rc)); } DEBUG_RET(); } void find_rfc822_headers(char** extra_mime_headers) { DEBUG_ENT("find_rfc822_headers"); char *headers = *extra_mime_headers; if (headers) { char *temp, *t; while ((temp = strstr(headers, "\n\n"))) { temp[1] = '\0'; t = header_get_field(headers, "\nContent-Type: "); if (t) { t++; DEBUG_EMAIL(("found content type header\n")); char *n = strchr(t, '\n'); char *s = strstr(t, ": "); char *e = strchr(t, ';'); if (!e || (e > n)) e = n; if (s && (s < e)) { s += 2; if (!strncasecmp(s, RFC822, e-s)) { headers = temp+2; // found rfc822 header DEBUG_EMAIL(("found 822 headers\n%s\n", headers)); break; } } } //DEBUG_EMAIL(("skipping to next block after\n%s\n", headers)); headers = temp+2; // skip to next chunk of headers } *extra_mime_headers = headers; } DEBUG_RET(); } void write_body_part(FILE* f_output, pst_string *body, char *mime, char *charset, char *boundary, pst_file* pst) { DEBUG_ENT("write_body_part"); if (body->is_utf8 && (strcasecmp("utf-8", charset))) { // try to convert to the specified charset since the target // is not utf-8, and the data came from a unicode (utf16) field // and is now in utf-8. size_t rc; DEBUG_EMAIL(("Convert %s utf-8 to %s\n", mime, charset)); pst_vbuf *newer = pst_vballoc(2); rc = pst_vb_utf8to8bit(newer, body->str, strlen(body->str), charset); if (rc == (size_t)-1) { // unable to convert, change the charset to utf8 free(newer->b); DEBUG_EMAIL(("Failed to convert %s utf-8 to %s\n", mime, charset)); charset = "utf-8"; } else { // null terminate the output string pst_vbgrow(newer, 1); newer->b[newer->dlen] = '\0'; free(body->str); body->str = newer->b; } free(newer); } removeCR(body->str); int base64 = test_base64(body->str); fprintf(f_output, "\n--%s\n", boundary); fprintf(f_output, "Content-Type: %s; charset=\"%s\"\n", mime, charset); if (base64) fprintf(f_output, "Content-Transfer-Encoding: base64\n"); fprintf(f_output, "\n"); if (base64) { char *enc = pst_base64_encode(body->str, strlen(body->str)); if (enc) { write_email_body(f_output, enc); fprintf(f_output, "\n"); free(enc); } } else { write_email_body(f_output, body->str); } DEBUG_RET(); } void write_normal_email(FILE* f_output, char f_name[], pst_item* item, int mode, int mode_MH, pst_file* pst, int save_rtf, char** extra_mime_headers) { char boundary[60]; char body_charset[60]; char body_report[60]; char sender[60]; int sender_known = 0; char *temp = NULL; time_t em_time; char *c_time; char *headers = NULL; int has_from, has_subject, has_to, has_cc, has_date, has_msgid; has_from = has_subject = has_to = has_cc = has_date = has_msgid = 0; DEBUG_ENT("write_normal_email"); pst_convert_utf8_null(item, &item->email->header); headers = (item->email->header.str) ? item->email->header.str : *extra_mime_headers; // setup default body character set and report type strncpy(body_charset, pst_default_charset(item), sizeof(body_charset)); body_charset[sizeof(body_charset)-1] = '\0'; body_report[0] = '\0'; // setup default sender pst_convert_utf8(item, &item->email->sender_address); if (item->email->sender_address.str && strchr(item->email->sender_address.str, '@')) { temp = item->email->sender_address.str; sender_known = 1; } else { temp = "MAILER-DAEMON"; } strncpy(sender, temp, sizeof(sender)); sender[sizeof(sender)-1] = '\0'; // convert the sent date if it exists, or set it to a fixed date if (item->email->sent_date) { em_time = pst_fileTimeToUnixTime(item->email->sent_date); c_time = ctime(&em_time); if (c_time) c_time[strlen(c_time)-1] = '\0'; //remove end \n else c_time = "Fri Dec 28 12:06:21 2001"; } else c_time= "Fri Dec 28 12:06:21 2001"; // create our MIME boundary here. snprintf(boundary, sizeof(boundary), "--boundary-LibPST-iamunique-%i_-_-", rand()); // we will always look at the headers to discover some stuff if (headers ) { char *t; removeCR(headers); temp = strstr(headers, "\n\n"); if (temp) { // cut off our real rfc822 headers here temp[1] = '\0'; // pointer to all the embedded MIME headers. // we use these to find the actual rfc822 headers for embedded message/rfc822 mime parts *extra_mime_headers = temp+2; DEBUG_EMAIL(("Found extra mime headers\n%s\n", temp+2)); } // Check if the headers have all the necessary fields header_has_field(headers, "\nFrom: ", &has_from); header_has_field(headers, "\nTo: ", &has_to); header_has_field(headers, "\nSubject: ", &has_subject); header_has_field(headers, "\nDate: ", &has_date); header_has_field(headers, "\nCC: ", &has_cc); header_has_field(headers, "\nMessage-Id: ", &has_msgid); // look for charset and report-type in Content-Type header t = header_get_field(headers, "\nContent-Type: "); header_get_subfield(t, "charset", body_charset, sizeof(body_charset)); header_get_subfield(t, "report-type", body_report, sizeof(body_report)); // derive a proper sender email address if (!sender_known) { t = header_get_field(headers, "\nFrom: "); if (t) { // assume address is on the first line, rather than on a continuation line t++; char *n = strchr(t, '\n'); char *s = strchr(t, '<'); char *e = strchr(t, '>'); if (s && e && n && (s < e) && (e < n)) { char save = *e; *e = '\0'; snprintf(sender, sizeof(sender), "%s", s+1); *e = save; } } } // Strip out the mime headers and some others that we don't want to emit header_strip_field(headers, "\nMicrosoft Mail Internet Headers"); header_strip_field(headers, "\nMIME-Version: "); header_strip_field(headers, "\nContent-Type: "); header_strip_field(headers, "\nContent-Transfer-Encoding: "); header_strip_field(headers, "\nContent-class: "); header_strip_field(headers, "\nX-MimeOLE: "); header_strip_field(headers, "\nBcc:"); header_strip_field(headers, "\nX-From_: "); } DEBUG_EMAIL(("About to print Header\n")); if (item && item->subject.str) { pst_convert_utf8(item, &item->subject); DEBUG_EMAIL(("item->subject = %s\n", item->subject.str)); } if (mode != MODE_SEPARATE) { // most modes need this separator line. // procmail produces this separator without the quotes around the // sender email address, but apparently some Mac email client needs // those quotes, and they don't seem to cause problems for anyone else. fprintf(f_output, "From \"%s\" %s\n", sender, c_time); } // print the supplied email headers if (headers) { int len; fprintf(f_output, "%s", headers); // make sure the headers end with a \n len = strlen(headers); if (!len || (headers[len-1] != '\n')) fprintf(f_output, "\n"); } // create required header fields that are not already written if (!has_from) { fprintf(f_output, "From: \"%s\" <%s>\n", item->email->outlook_sender_name.str, sender); } if (!has_subject) { if (item->subject.str) { fprintf(f_output, "Subject: %s\n", item->subject.str); } else { fprintf(f_output, "Subject: \n"); } } if (!has_to && item->email->sentto_address.str) { pst_convert_utf8(item, &item->email->sentto_address); fprintf(f_output, "To: %s\n", item->email->sentto_address.str); } if (!has_cc && item->email->cc_address.str) { pst_convert_utf8(item, &item->email->cc_address); fprintf(f_output, "Cc: %s\n", item->email->cc_address.str); } if (!has_date && item->email->sent_date) { char c_time[C_TIME_SIZE]; strftime(c_time, C_TIME_SIZE, "%a, %d %b %Y %H:%M:%S %z", gmtime(&em_time)); fprintf(f_output, "Date: %s\n", c_time); } if (!has_msgid && item->email->messageid.str) { pst_convert_utf8(item, &item->email->messageid); fprintf(f_output, "Message-Id: %s\n", item->email->messageid.str); } // add forensic headers to capture some .pst stuff that is not really // needed or used by mail clients pst_convert_utf8_null(item, &item->email->sender_address); if (item->email->sender_address.str && !strchr(item->email->sender_address.str, '@') && strcmp(item->email->sender_address.str, ".")) { fprintf(f_output, "X-libpst-forensic-sender: %s\n", item->email->sender_address.str); } if (item->email->bcc_address.str) { pst_convert_utf8(item, &item->email->bcc_address); fprintf(f_output, "X-libpst-forensic-bcc: %s\n", item->email->bcc_address.str); } // add our own mime headers fprintf(f_output, "MIME-Version: 1.0\n"); if (body_report[0] != '\0') { // multipart/report for DSN/MDN reports fprintf(f_output, "Content-Type: multipart/report; report-type=%s;\n\tboundary=\"%s\"\n", body_report, boundary); } else if (item->attach || (item->email->rtf_compressed.data && save_rtf) || item->email->encrypted_body.data || item->email->encrypted_htmlbody.data) { // use multipart/mixed if we have attachments fprintf(f_output, "Content-Type: multipart/mixed;\n\tboundary=\"%s\"\n", boundary); } else { // else use multipart/alternative fprintf(f_output, "Content-Type: multipart/alternative;\n\tboundary=\"%s\"\n", boundary); } fprintf(f_output, "\n"); // end of headers, start of body // now dump the body parts if (item->body.str) { write_body_part(f_output, &item->body, "text/plain", body_charset, boundary, pst); } if ((item->email->report_text.str) && (body_report[0] != '\0')) { write_body_part(f_output, &item->email->report_text, "text/plain", body_charset, boundary, pst); fprintf(f_output, "\n"); } if (item->email->htmlbody.str) { find_html_charset(item->email->htmlbody.str, body_charset, sizeof(body_charset)); write_body_part(f_output, &item->email->htmlbody, "text/html", body_charset, boundary, pst); } if (item->email->rtf_compressed.data && save_rtf) { pst_item_attach* attach = (pst_item_attach*)pst_malloc(sizeof(pst_item_attach)); DEBUG_EMAIL(("Adding RTF body as attachment\n")); memset(attach, 0, sizeof(pst_item_attach)); attach->next = item->attach; item->attach = attach; attach->data.data = pst_lzfu_decompress(item->email->rtf_compressed.data, item->email->rtf_compressed.size, &attach->data.size); attach->filename2.str = strdup(RTF_ATTACH_NAME); attach->filename2.is_utf8 = 1; attach->mimetype.str = strdup(RTF_ATTACH_TYPE); attach->mimetype.is_utf8 = 1; } if (item->email->encrypted_body.data || item->email->encrypted_htmlbody.data) { // if either the body or htmlbody is encrypted, add them as attachments if (item->email->encrypted_body.data) { pst_item_attach* attach = (pst_item_attach*)pst_malloc(sizeof(pst_item_attach)); DEBUG_EMAIL(("Adding Encrypted Body as attachment\n")); attach = (pst_item_attach*) pst_malloc(sizeof(pst_item_attach)); memset(attach, 0, sizeof(pst_item_attach)); attach->next = item->attach; item->attach = attach; attach->data.data = item->email->encrypted_body.data; attach->data.size = item->email->encrypted_body.size; item->email->encrypted_body.data = NULL; } if (item->email->encrypted_htmlbody.data) { pst_item_attach* attach = (pst_item_attach*)pst_malloc(sizeof(pst_item_attach)); DEBUG_EMAIL(("Adding encrypted HTML body as attachment\n")); attach = (pst_item_attach*) pst_malloc(sizeof(pst_item_attach)); memset(attach, 0, sizeof(pst_item_attach)); attach->next = item->attach; item->attach = attach; attach->data.data = item->email->encrypted_htmlbody.data; attach->data.size = item->email->encrypted_htmlbody.size; item->email->encrypted_htmlbody.data = NULL; } write_email_body(f_output, "The body of this email is encrypted. This isn't supported yet, but the body is now an attachment\n"); } // other attachments { pst_item_attach* attach; int attach_num = 0; for (attach = item->attach; attach; attach = attach->next) { pst_convert_utf8_null(item, &attach->filename1); pst_convert_utf8_null(item, &attach->filename2); pst_convert_utf8_null(item, &attach->mimetype); DEBUG_EMAIL(("Attempting Attachment encoding\n")); if (!attach->data.data && attach->mimetype.str && !strcmp(attach->mimetype.str, RFC822)) { DEBUG_EMAIL(("seem to have special embedded message attachment\n")); find_rfc822_headers(extra_mime_headers); write_embedded_message(f_output, attach, boundary, pst, extra_mime_headers); } else if (attach->data.data || attach->i_id) { if (mode == MODE_SEPARATE && !mode_MH) write_separate_attachment(f_name, attach, ++attach_num, pst); else write_inline_attachment(f_output, attach, boundary, pst); } } } fprintf(f_output, "\n--%s--\n\n\n", boundary); DEBUG_RET(); } void write_vcard(FILE* f_output, pst_item *item, pst_item_contact* contact, char comment[]) { // We can only call rfc escape once per printf, since the second call // may free the buffer returned by the first call. // I had tried to place those into a single printf - Carl. DEBUG_ENT("write_vcard"); // make everything utf8 pst_convert_utf8_null(item, &contact->fullname); pst_convert_utf8_null(item, &contact->surname); pst_convert_utf8_null(item, &contact->first_name); pst_convert_utf8_null(item, &contact->middle_name); pst_convert_utf8_null(item, &contact->display_name_prefix); pst_convert_utf8_null(item, &contact->suffix); pst_convert_utf8_null(item, &contact->nickname); pst_convert_utf8_null(item, &contact->address1); pst_convert_utf8_null(item, &contact->address2); pst_convert_utf8_null(item, &contact->address3); pst_convert_utf8_null(item, &contact->home_po_box); pst_convert_utf8_null(item, &contact->home_street); pst_convert_utf8_null(item, &contact->home_city); pst_convert_utf8_null(item, &contact->home_state); pst_convert_utf8_null(item, &contact->home_postal_code); pst_convert_utf8_null(item, &contact->home_country); pst_convert_utf8_null(item, &contact->home_address); pst_convert_utf8_null(item, &contact->business_po_box); pst_convert_utf8_null(item, &contact->business_street); pst_convert_utf8_null(item, &contact->business_city); pst_convert_utf8_null(item, &contact->business_state); pst_convert_utf8_null(item, &contact->business_postal_code); pst_convert_utf8_null(item, &contact->business_country); pst_convert_utf8_null(item, &contact->business_address); pst_convert_utf8_null(item, &contact->other_po_box); pst_convert_utf8_null(item, &contact->other_street); pst_convert_utf8_null(item, &contact->other_city); pst_convert_utf8_null(item, &contact->other_state); pst_convert_utf8_null(item, &contact->other_postal_code); pst_convert_utf8_null(item, &contact->other_country); pst_convert_utf8_null(item, &contact->other_address); pst_convert_utf8_null(item, &contact->business_fax); pst_convert_utf8_null(item, &contact->business_phone); pst_convert_utf8_null(item, &contact->business_phone2); pst_convert_utf8_null(item, &contact->car_phone); pst_convert_utf8_null(item, &contact->home_fax); pst_convert_utf8_null(item, &contact->home_phone); pst_convert_utf8_null(item, &contact->home_phone2); pst_convert_utf8_null(item, &contact->isdn_phone); pst_convert_utf8_null(item, &contact->mobile_phone); pst_convert_utf8_null(item, &contact->other_phone); pst_convert_utf8_null(item, &contact->pager_phone); pst_convert_utf8_null(item, &contact->primary_fax); pst_convert_utf8_null(item, &contact->primary_phone); pst_convert_utf8_null(item, &contact->radio_phone); pst_convert_utf8_null(item, &contact->telex); pst_convert_utf8_null(item, &contact->job_title); pst_convert_utf8_null(item, &contact->profession); pst_convert_utf8_null(item, &contact->assistant_name); pst_convert_utf8_null(item, &contact->assistant_phone); pst_convert_utf8_null(item, &contact->company_name); // the specification I am following is (hopefully) RFC2426 vCard Mime Directory Profile fprintf(f_output, "BEGIN:VCARD\n"); fprintf(f_output, "FN:%s\n", pst_rfc2426_escape(contact->fullname.str)); //fprintf(f_output, "N:%s;%s;%s;%s;%s\n", fprintf(f_output, "N:%s;", (!contact->surname.str) ? "" : pst_rfc2426_escape(contact->surname.str)); fprintf(f_output, "%s;", (!contact->first_name.str) ? "" : pst_rfc2426_escape(contact->first_name.str)); fprintf(f_output, "%s;", (!contact->middle_name.str) ? "" : pst_rfc2426_escape(contact->middle_name.str)); fprintf(f_output, "%s;", (!contact->display_name_prefix.str) ? "" : pst_rfc2426_escape(contact->display_name_prefix.str)); fprintf(f_output, "%s\n", (!contact->suffix.str) ? "" : pst_rfc2426_escape(contact->suffix.str)); if (contact->nickname.str) fprintf(f_output, "NICKNAME:%s\n", pst_rfc2426_escape(contact->nickname.str)); if (contact->address1.str) fprintf(f_output, "EMAIL:%s\n", pst_rfc2426_escape(contact->address1.str)); if (contact->address2.str) fprintf(f_output, "EMAIL:%s\n", pst_rfc2426_escape(contact->address2.str)); if (contact->address3.str) fprintf(f_output, "EMAIL:%s\n", pst_rfc2426_escape(contact->address3.str)); if (contact->birthday) fprintf(f_output, "BDAY:%s\n", pst_rfc2425_datetime_format(contact->birthday)); if (contact->home_address.str) { //fprintf(f_output, "ADR;TYPE=home:%s;%s;%s;%s;%s;%s;%s\n", fprintf(f_output, "ADR;TYPE=home:%s;", (!contact->home_po_box.str) ? "" : pst_rfc2426_escape(contact->home_po_box.str)); fprintf(f_output, "%s;", ""); // extended Address fprintf(f_output, "%s;", (!contact->home_street.str) ? "" : pst_rfc2426_escape(contact->home_street.str)); fprintf(f_output, "%s;", (!contact->home_city.str) ? "" : pst_rfc2426_escape(contact->home_city.str)); fprintf(f_output, "%s;", (!contact->home_state.str) ? "" : pst_rfc2426_escape(contact->home_state.str)); fprintf(f_output, "%s;", (!contact->home_postal_code.str) ? "" : pst_rfc2426_escape(contact->home_postal_code.str)); fprintf(f_output, "%s\n", (!contact->home_country.str) ? "" : pst_rfc2426_escape(contact->home_country.str)); fprintf(f_output, "LABEL;TYPE=home:%s\n", pst_rfc2426_escape(contact->home_address.str)); } if (contact->business_address.str) { //fprintf(f_output, "ADR;TYPE=work:%s;%s;%s;%s;%s;%s;%s\n", fprintf(f_output, "ADR;TYPE=work:%s;", (!contact->business_po_box.str) ? "" : pst_rfc2426_escape(contact->business_po_box.str)); fprintf(f_output, "%s;", ""); // extended Address fprintf(f_output, "%s;", (!contact->business_street.str) ? "" : pst_rfc2426_escape(contact->business_street.str)); fprintf(f_output, "%s;", (!contact->business_city.str) ? "" : pst_rfc2426_escape(contact->business_city.str)); fprintf(f_output, "%s;", (!contact->business_state.str) ? "" : pst_rfc2426_escape(contact->business_state.str)); fprintf(f_output, "%s;", (!contact->business_postal_code.str) ? "" : pst_rfc2426_escape(contact->business_postal_code.str)); fprintf(f_output, "%s\n", (!contact->business_country.str) ? "" : pst_rfc2426_escape(contact->business_country.str)); fprintf(f_output, "LABEL;TYPE=work:%s\n", pst_rfc2426_escape(contact->business_address.str)); } if (contact->other_address.str) { //fprintf(f_output, "ADR;TYPE=postal:%s;%s;%s;%s;%s;%s;%s\n", fprintf(f_output, "ADR;TYPE=postal:%s;",(!contact->other_po_box.str) ? "" : pst_rfc2426_escape(contact->other_po_box.str)); fprintf(f_output, "%s;", ""); // extended Address fprintf(f_output, "%s;", (!contact->other_street.str) ? "" : pst_rfc2426_escape(contact->other_street.str)); fprintf(f_output, "%s;", (!contact->other_city.str) ? "" : pst_rfc2426_escape(contact->other_city.str)); fprintf(f_output, "%s;", (!contact->other_state.str) ? "" : pst_rfc2426_escape(contact->other_state.str)); fprintf(f_output, "%s;", (!contact->other_postal_code.str) ? "" : pst_rfc2426_escape(contact->other_postal_code.str)); fprintf(f_output, "%s\n", (!contact->other_country.str) ? "" : pst_rfc2426_escape(contact->other_country.str)); fprintf(f_output, "LABEL;TYPE=postal:%s\n", pst_rfc2426_escape(contact->other_address.str)); } if (contact->business_fax.str) fprintf(f_output, "TEL;TYPE=work,fax:%s\n", pst_rfc2426_escape(contact->business_fax.str)); if (contact->business_phone.str) fprintf(f_output, "TEL;TYPE=work,voice:%s\n", pst_rfc2426_escape(contact->business_phone.str)); if (contact->business_phone2.str) fprintf(f_output, "TEL;TYPE=work,voice:%s\n", pst_rfc2426_escape(contact->business_phone2.str)); if (contact->car_phone.str) fprintf(f_output, "TEL;TYPE=car,voice:%s\n", pst_rfc2426_escape(contact->car_phone.str)); if (contact->home_fax.str) fprintf(f_output, "TEL;TYPE=home,fax:%s\n", pst_rfc2426_escape(contact->home_fax.str)); if (contact->home_phone.str) fprintf(f_output, "TEL;TYPE=home,voice:%s\n", pst_rfc2426_escape(contact->home_phone.str)); if (contact->home_phone2.str) fprintf(f_output, "TEL;TYPE=home,voice:%s\n", pst_rfc2426_escape(contact->home_phone2.str)); if (contact->isdn_phone.str) fprintf(f_output, "TEL;TYPE=isdn:%s\n", pst_rfc2426_escape(contact->isdn_phone.str)); if (contact->mobile_phone.str) fprintf(f_output, "TEL;TYPE=cell,voice:%s\n", pst_rfc2426_escape(contact->mobile_phone.str)); if (contact->other_phone.str) fprintf(f_output, "TEL;TYPE=msg:%s\n", pst_rfc2426_escape(contact->other_phone.str)); if (contact->pager_phone.str) fprintf(f_output, "TEL;TYPE=pager:%s\n", pst_rfc2426_escape(contact->pager_phone.str)); if (contact->primary_fax.str) fprintf(f_output, "TEL;TYPE=fax,pref:%s\n", pst_rfc2426_escape(contact->primary_fax.str)); if (contact->primary_phone.str) fprintf(f_output, "TEL;TYPE=phone,pref:%s\n", pst_rfc2426_escape(contact->primary_phone.str)); if (contact->radio_phone.str) fprintf(f_output, "TEL;TYPE=pcs:%s\n", pst_rfc2426_escape(contact->radio_phone.str)); if (contact->telex.str) fprintf(f_output, "TEL;TYPE=bbs:%s\n", pst_rfc2426_escape(contact->telex.str)); if (contact->job_title.str) fprintf(f_output, "TITLE:%s\n", pst_rfc2426_escape(contact->job_title.str)); if (contact->profession.str) fprintf(f_output, "ROLE:%s\n", pst_rfc2426_escape(contact->profession.str)); if (contact->assistant_name.str || contact->assistant_phone.str) { fprintf(f_output, "AGENT:BEGIN:VCARD\n"); if (contact->assistant_name.str) fprintf(f_output, "FN:%s\n", pst_rfc2426_escape(contact->assistant_name.str)); if (contact->assistant_phone.str) fprintf(f_output, "TEL:%s\n", pst_rfc2426_escape(contact->assistant_phone.str)); } if (contact->company_name.str) fprintf(f_output, "ORG:%s\n", pst_rfc2426_escape(contact->company_name.str)); if (comment) fprintf(f_output, "NOTE:%s\n", pst_rfc2426_escape(comment)); fprintf(f_output, "VERSION: 3.0\n"); fprintf(f_output, "END:VCARD\n\n"); DEBUG_RET(); } void write_appointment(FILE* f_output, pst_item *item, pst_item_appointment* appointment, FILETIME* create_date, FILETIME* modify_date) { // make everything utf8 pst_convert_utf8_null(item, &item->subject); pst_convert_utf8_null(item, &item->body); pst_convert_utf8_null(item, &appointment->location); fprintf(f_output, "BEGIN:VEVENT\n"); if (create_date) fprintf(f_output, "CREATED:%s\n", pst_rfc2445_datetime_format(create_date)); if (modify_date) fprintf(f_output, "LAST-MOD:%s\n", pst_rfc2445_datetime_format(modify_date)); if (item->subject.str) fprintf(f_output, "SUMMARY:%s\n", pst_rfc2426_escape(item->subject.str)); if (item->body.str) fprintf(f_output, "DESCRIPTION:%s\n", pst_rfc2426_escape(item->body.str)); if (appointment && appointment->start) fprintf(f_output, "DTSTART;VALUE=DATE-TIME:%s\n", pst_rfc2445_datetime_format(appointment->start)); if (appointment && appointment->end) fprintf(f_output, "DTEND;VALUE=DATE-TIME:%s\n", pst_rfc2445_datetime_format(appointment->end)); if (appointment && appointment->location.str) fprintf(f_output, "LOCATION:%s\n", pst_rfc2426_escape(appointment->location.str)); if (appointment) { switch (appointment->showas) { case PST_FREEBUSY_TENTATIVE: fprintf(f_output, "STATUS:TENTATIVE\n"); break; case PST_FREEBUSY_FREE: // mark as transparent and as confirmed fprintf(f_output, "TRANSP:TRANSPARENT\n"); case PST_FREEBUSY_BUSY: case PST_FREEBUSY_OUT_OF_OFFICE: fprintf(f_output, "STATUS:CONFIRMED\n"); break; } switch (appointment->label) { case PST_APP_LABEL_NONE: fprintf(f_output, "CATEGORIES:NONE\n"); break; case PST_APP_LABEL_IMPORTANT: fprintf(f_output, "CATEGORIES:IMPORTANT\n"); break; case PST_APP_LABEL_BUSINESS: fprintf(f_output, "CATEGORIES:BUSINESS\n"); break; case PST_APP_LABEL_PERSONAL: fprintf(f_output, "CATEGORIES:PERSONAL\n"); break; case PST_APP_LABEL_VACATION: fprintf(f_output, "CATEGORIES:VACATION\n"); break; case PST_APP_LABEL_MUST_ATTEND: fprintf(f_output, "CATEGORIES:MUST-ATTEND\n"); break; case PST_APP_LABEL_TRAVEL_REQ: fprintf(f_output, "CATEGORIES:TRAVEL-REQUIRED\n"); break; case PST_APP_LABEL_NEEDS_PREP: fprintf(f_output, "CATEGORIES:NEEDS-PREPARATION\n"); break; case PST_APP_LABEL_BIRTHDAY: fprintf(f_output, "CATEGORIES:BIRTHDAY\n"); break; case PST_APP_LABEL_ANNIVERSARY: fprintf(f_output, "CATEGORIES:ANNIVERSARY\n"); break; case PST_APP_LABEL_PHONE_CALL: fprintf(f_output, "CATEGORIES:PHONE-CALL\n"); break; } } fprintf(f_output, "END:VEVENT\n\n"); } void create_enter_dir(struct file_ll* f, pst_item *item) { pst_convert_utf8(item, &item->file_as); f->item_count = 0; f->skip_count = 0; f->type = item->type; f->stored_count = (item->folder) ? item->folder->item_count : 0; DEBUG_ENT("create_enter_dir"); if (mode == MODE_KMAIL) f->name = mk_kmail_dir(item->file_as.str); else if (mode == MODE_RECURSE) f->name = mk_recurse_dir(item->file_as.str, f->type); else if (mode == MODE_SEPARATE) { // do similar stuff to recurse here. mk_separate_dir(item->file_as.str); f->name = (char*) pst_malloc(10); memset(f->name, 0, 10); } else { f->name = (char*) pst_malloc(strlen(item->file_as.str)+strlen(OUTPUT_TEMPLATE)+1); sprintf(f->name, OUTPUT_TEMPLATE, item->file_as.str); } f->dname = (char*) pst_malloc(strlen(item->file_as.str)+1); strcpy(f->dname, item->file_as.str); if (overwrite != 1) { int x = 0; char *temp = (char*) pst_malloc (strlen(f->name)+10); //enough room for 10 digits sprintf(temp, "%s", f->name); check_filename(temp); while ((f->output = fopen(temp, "r"))) { DEBUG_MAIN(("need to increase filename because one already exists with that name\n")); DEBUG_MAIN(("- increasing it to %s%d\n", f->name, x)); x++; sprintf(temp, "%s%08d", f->name, x); DEBUG_MAIN(("- trying \"%s\"\n", f->name)); if (x == 99999999) { DIE(("create_enter_dir: Why can I not create a folder %s? I have tried %i extensions...\n", f->name, x)); } fclose(f->output); } if (x > 0) { //then the f->name should change free (f->name); f->name = temp; } else { free(temp); } } DEBUG_MAIN(("f->name = %s\nitem->folder_name = %s\n", f->name, item->file_as.str)); if (mode != MODE_SEPARATE) { check_filename(f->name); if (!(f->output = fopen(f->name, "w"))) { DIE(("create_enter_dir: Could not open file \"%s\" for write\n", f->name)); } } DEBUG_RET(); } void close_enter_dir(struct file_ll *f) { DEBUG_MAIN(("main: processed item count for folder %s is %i, skipped %i, total %i \n", f->dname, f->item_count, f->skip_count, f->stored_count)); if (output_mode != OUTPUT_QUIET) printf("\t\"%s\" - %i items done, %i items skipped.\n", f->dname, f->item_count, f->skip_count); if (f->output) fclose(f->output); free(f->name); free(f->dname); if (mode == MODE_KMAIL) close_kmail_dir(); else if (mode == MODE_RECURSE) close_recurse_dir(); else if (mode == MODE_SEPARATE) close_separate_dir(); } diff --git a/xml/libpst.in b/xml/libpst.in index d46d989..649eee8 100644 --- a/xml/libpst.in +++ b/xml/libpst.in @@ -1,2152 +1,2145 @@ @PACKAGE@ Utilities - Version @VERSION@ Packages The various source and binary packages are available at http://www.five-ten-sg.com/@PACKAGE@/packages/. The most recent documentation is available at http://www.five-ten-sg.com/@PACKAGE@/. The most recent developer documentation for the shared library is available at http://www.five-ten-sg.com/@PACKAGE@/devel/. A Mercurial source code repository for this project is available at http://hg.five-ten-sg.com/@PACKAGE@/. This version can now convert both 32 bit Outlook files (pre 2003), and the 64 bit Outlook 2003 pst files. Utilities are supplied to convert email messages to both mbox and MH mailbox formats, and to DII load file format for use with many of the CT Summation products. Contacts can be converted to a simple list, to vcard format, or to ldif format for import to an LDAP server. The libpff project has some excellent documentation of the pst file format. 2009-04-15 readpst 1 readpst @VERSION@ readpst convert PST (MS Outlook Personal Folders) files to mbox and other formats Synopsis readpst - pstfile Description readpst is a program that can read an Outlook PST (Personal Folders) file and convert it into an mbox file, a format suitable for KMail, a recursive mbox structure, or separate emails. Options - - -C - - Decrypt the entire pst file and dump it to stdout. - - -D Include deleted items in the output. -M Output messages in MH format as separate files. This will create folders as named in the PST file, and will put each email together with any attachments into its own file. These files will be numbered from 1 to n with no leading zeros. -S Output messages into separate files. This will create folders as named in the PST file, and will put each email in its own file. These files will be numbered from 1 increasing in intervals of 1 (ie 1, 2, 3, ...). Any attachments are saved alongside each email as XXXXXXXXX-attach1, XXXXXXXXX-attach2 and so on, or with the name of the attachment if one is present. -V Show program version and exit. -b Do not save the attachments for the RTF format of the email body. -c format Set the Contact output mode. Use -cv for vcard format or -cl for an email list. -d debug-file Specify name of debug log file. The log file is not an ascii file, it is a binary file readable by readpstlog. -h Show summary of options and exit. -k Changes the output format to KMail. -o output-directory Specifies the output directory. The directory must already exist, and is entered after the PST file is opened, but before any processing of files commences. -q Changes to silent mode. No feedback is printed to the screen, except for error messages. -r Changes the output format to Recursive. This will create folders as named in the PST file, and will put all emails in a file called "mbox" inside each folder. These files are then compatible with all mbox-compatible email clients. -w Overwrite any previous output files. Beware: When used with the -S switch, this will remove all files from the target folder before writing. This is to keep the count of emails and attachments correct. See Also readpstlog 1 Author This manual page was originally written by Dave Smith <dave.s@earthcorp.com>, and updated by Joe Nahmias <joe@nahmias.net> for the Debian GNU/Linux system (but may be used by others). It was subsequently updated by Brad Hards <bradh@frogmouth.net>, and converted to xml format by Carl Byington <carl@five-ten-sg.com>. Copyright Copyright (C) 2002 by David Smith <dave.s@earthcorp.com>. XML version Copyright (C) 2008 by 510 Software Group <carl@five-ten-sg.com>. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2009-04-15 lspst 1 lspst @VERSION@ lspst list PST (MS Outlook Personal Folders) file data Synopsis lspst pstfile Options -V Show program version and exit. -d debug-file Specify name of debug log file. The log file is not an ascii file, it is a binary file readable by readpstlog. -h Show summary of options and exit. Description lspst is a program that can read an Outlook PST (Personal Folders) file and produce a simple listing of the data (contacts, email subjects, etc). See Also readpstlog 1 Author lspst was written by Joe Nahmias <joe@nahmias.net> based on readpst. This man page was written by 510 Software Group <carl@five-ten-sg.com>. Copyright Copyright (C) 2004 by Joe Nahmias <joe@nahmias.net>. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2009-04-15 readpstlog 1 readpstlog @VERSION@ readpstlog convert a readpst logfile to text format Synopsis readpstlog logfile Description readpstlog is a program that converts the binary logfile generated by readpst to a more desirable text format. Options -f format Sets the format of the text log output. Currently, the only valid output formats are T, for single line text, D for the default default multi line format, and I for an indented style with single line text. -t include-types Print only the specified types of log messages. Types are specified in a comma-delimited list (e.g. 3,10,5,6). -x exclude-types Exclude the specified types of log messages. Types are specified in a comma-delimited list (e.g. 3,10,5,6). Message Types readpstlog understands the following types of log messages: 1 File accesses 2 Index accesses 3 New email found 4 Warnings 5 Read accesses 6 Informational messages 7 Main function calls 8 Decrypting calls 9 Function entries 10 Function exits 11 HexDump calls Author This manual page was written by Joe Nahmias <joe@nahmias.net> for the Debian GNU/Linux system (but may be used by others). It was converted to xml format by Carl Byington <carl@five-ten-sg.com>. Copyright Copyright (C) 2002 by David Smith <dave.s@earthcorp.com>. XML version Copyright (C) 2008 by 510 Software Group <carl@five-ten-sg.com>. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2009-04-15 pst2ldif 1 pst2ldif @VERSION@ pst2ldif extract contacts from a MS Outlook .pst file in .ldif format Synopsis pst2ldif pstfilename Options -V Show program version. Subsequent options are then ignored. -b ldap-base Sets the ldap base value used in the dn records. You probably want to use something like "o=organization, c=US". -c class Sets the objectClass values for the contact items. This class needs to be defined in the schema used by your LDAP server, and at a minimum it must contain the ldap attributes given below. This option may be specified multiple times to generate entries with multiple object classes. -d debug-file Specify name of debug log file. The log file is not an ascii file, it is a binary file readable by readpstlog. -l extra-line Specify an extra line to be added to each ldap entry. This option may be specified multiple times to add multiple lines to each ldap entry. -o Use the old ldap schema, rather than the default new ldap schema. The old schema generates multiple postalAddress attributes for a single entry. The new schema generates a single postalAddress (and homePostalAddress when available) attribute with $ delimiters as specified in RFC4517. Using the old schema also generates two extra leading entries, one for "dn:ldap base", and one for "dn: cn=root, ldap base". -h Show summary of options. Subsequent options are then ignored. Description pst2ldif reads the contact information from a MS Outlook .pst file and produces a .ldif file that may be used to import those contacts into an LDAP database. The following ldap attributes are generated for the old ldap schema: cn givenName sn personalTitle company mail postalAddress l st postalCode c homePhone telephoneNumber facsimileTelephoneNumber mobile description The following attributes are generated for the new ldap schema: cn givenName sn title o mail postalAddress homePostalAddress l st postalCode c homePhone telephoneNumber facsimileTelephoneNumber mobile description labeledURI Copyright Copyright (C) 2008 by 510 Software Group <carl@five-ten-sg.com> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2009-04-15 pst2dii 1 pst2dii @VERSION@ pst2dii extract email messages from a MS Outlook .pst file in DII load format Synopsis pst2dii pstfilename Options -B bates-prefix Sets the bates prefix string. The bates sequence number is appended to this string, and printed on each page. -O dii-output-file Name of the output DII load file. -V Show program version. Subsequent options are then ignored. -b bates-number Starting bates sequence number. The default is zero. -c bates-color Font color for the bates stamp on each page, specified as 6 hex digits as rrggbb values. The default is ff0000 for bright red. -d debug-file Specify name of debug log file. The log file is not an ascii file, it is a binary file readable by readpstlog. -f ttf-font-file Specify name of a true type font file. This should be a fixed pitch font. -h Show summary of options. Subsequent options are then ignored. -o output-directory Specifies the output directory. The directory must already exist. Description pst2dii reads the email messages from a MS Outlook .pst file and produces a DII load file that may be used to import message summaries into a Summation DII system. The DII output file contains references to the image and attachment files in the output directory. Copyright Copyright (C) 2008 by 510 Software Group <carl@five-ten-sg.com> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2009-04-15 outlook.pst 5 outlook.pst format of MS Outlook .pst file Synopsis outlook.pst Overview Low level or primitive items in a .pst file are identified by an I_ID value. Higher level or composite items in a .pst file are identified by a D_ID value. There are two separate b-trees indexed by these I_ID and D_ID values. Starting with Outlook 2003, the file format changed from one with 32 bit pointers, to one with 64 bit pointers. We describe both formats here. 32 bit File Header The 32 bit file header is located at offset 0 in the .pst file. We only support index types 0x0e, 0x0f, 0x15, and 0x17, and encryption types 0x00, 0x01 and 0x02. Index type 0x0e is the older 32 bit Outlook format. Index type 0x0f seems to be rare, and so far the data seems to be identical to that in type 0x0e files. Index type 0x17 is the newer 64 bit Outlook format. Index type 0x15 seems to be rare, and according to the libpff project should have the same format as type 0x17 files. It was found in a 64-bit pst file created by Visual Recovery. It may be that index types less than 0x10 are 32 bit, and index types greater than or equal to 0x10 are 64 bit, and the low order four bits of the index type is some subtype or minor version number. Encryption type 0x00 is no encryption, type 0x01 is "compressible" encryption which is a simple substitution cipher, and type 0x02 is "strong" encryption, which is a simple three rotor Enigma cipher from WWII. offsetIndex1 is the file offset of the root of the index1 b-tree, which contains (I_ID, offset, size, unknown) tuples for each item in the file. backPointer1 is the value that should appear in the parent pointer of that root node. offsetIndex2 is the file offset of the root of the index2 b-tree, which contains (D_ID, DESC-I_ID, TREE-I_ID, PARENT-D_ID) tuples for each item in the file. backPointer2 is the value that should appear in the parent pointer of that root node. 64 bit File Header The 64 bit file header is located at offset 0 in the .pst file. 32 bit Index 1 Node The 32 bit index1 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 12 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (I_ID, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and I_ID is the lowest I_ID value in the subtree. 64 bit Index 1 Node The 64 bit index1 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 24 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (I_ID, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and I_ID is the lowest I_ID value in the subtree. 32 bit Index 1 Leaf Node The 32 bit index1 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 12 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (I_ID, offset, size, unknown) The two low order bits of the I_ID value seem to be flags. I have never seen a case with bit zero set. Bit one indicates that the item is not encrypted. Note that references to these I_ID values elsewhere may have the low order bit set (and I don't know what that means), but when we do the search in this tree we need to clear that bit so that we can find the correct item. 64 bit Index 1 Leaf Node The 64 bit index1 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 24 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (I_ID, offset, size, unknown) The two low order bits of the I_ID value seem to be flags. I have never seen a case with bit zero set. Bit one indicates that the item is not encrypted. Note that references to these I_ID values elsewhere may have the low order bit set (and I don't know what that means), but when we do the search in this tree we need to clear that bit so that we can find the correct item. 32 bit Index 2 Node The 32 bit index2 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 12 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (D_ID, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and D_ID is the lowest D_ID value in the subtree. 64 bit Index 2 Node The 64 bit index2 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 24 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (D_ID, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and D_ID is the lowest D_ID value in the subtree. 32 bit Index 2 Leaf Node The 32 bit index2 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 16 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (D_ID, DESC-I_ID, TREE-I_ID, PARENT-D_ID) The DESC-I_ID points to the main data for this item (Associated Descriptor Items 0x7cec, 0xbcec, or 0x0101) via the index1 tree. The TREE-I_ID is zero or points to an Associated Tree Item 0x0002 via the index1 tree. The PARENT-D_ID points to the parent of this item in this index2 tree. 64 bit Index 2 Leaf Node The 64 bit index2 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 32 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (D_ID, DESC-I_ID, TREE-I_ID, PARENT-D_ID) The DESC-I_ID points to the main data for this item (Associated Descriptor Items 0x7cec, 0xbcec, or 0x0101) via the index1 tree. The TREE-I_ID is zero or points to an Associated Tree Item 0x0002 via the index1 tree. The PARENT-D_ID points to the parent of this item in this index2 tree. 32 bit Associated Tree Item 0x0002 A D_ID value may point to an entry in the index2 tree with a non-zero TREE-I_ID which points to this descriptor block via the index1 tree. It maps local ID2 values (referenced in the main data for the original D_ID item) to I_ID values. This descriptor block contains triples of (ID2, I_ID, CHILD-I_ID) where the local ID2 data can be found via I_ID, and CHILD-I_ID is either zero or it points to another Associated Tree Item via the index1 tree. In the above 32 bit leaf node, we have a tuple of (0x61, 0x02a82c, 0x02a836, 0) 0x02a836 is the I_ID of the associated tree, and we can lookup that I_ID value in the index1 b-tree to find the (offset,size) of the data in the .pst file. 64 bit Associated Tree Item 0x0002 This descriptor block contains a tree that maps local ID2 values to I_ID entries, similar to the 32 bit version described above. Associated Descriptor Item 0xbcec Contains information about the item, which may be email, contact, or other outlook types. In the above leaf node, we have a tuple of (0x21, 0x00e638, 0, 0) 0x00e638 is the I_ID of the associated descriptor, and we can lookup that I_ID value in the index1 b-tree to find the (offset,size) of the data in the .pst file. This descriptor is eventually decoded to a list of MAPI elements. Note the signature of 0xbcec. There are other descriptor block formats with other signatures. Note the indexOffset of 0x013c - starting at that position in the descriptor block, we have an array of two byte integers. The first integer (0x000b) is a (count-1) of the number of overlapping pairs following the count. The first pair is (0, 0xc), the next pair is (0xc, 0x14) and the last (12th) pair is (0x123, 0x13b). These pairs are (start,end+1) offsets of items in this block. So we have count+2 integers following the count value. Note the b5offset of 0x0020, which is a type that I will call an index reference. Such index references have at least two different forms, and may point to data either in this block, or in some other block. External pointer references have the low order 4 bits all set, and are ID2 values that can be used to fetch data. This value of 0x0020 is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0002, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xc, 0x14) pair. So far we have only described internal index references where the high order 16 bits are zero. That suffices for single descriptor blocks. But in the case of the type 0x0101 descriptor block, we have an array of subblocks. In this case, the high order 16 bits of an internal index reference are used to select the subblock. Each subblock starts with a 16 bit indexOffset which points to the count and array of 16 bit integer pairs which are offsets in the current subblock. Finally, we have the offset and size of the "b5" block located at offset 0xc with a size of 8 bytes in this descriptor block. The "b5" block has the following format: Note the descoffset of 0x0040, which again is an index reference. In this case, it is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0004, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0x14, 0x7c) pair. The datasize (6) plus the b5 code (02) gives the size of the entries, in this case 8 bytes. We now have the offset 0x14 of the descriptor array, composed of 8 byte entries that describe MAPI elements. Each descriptor entry has the following format: For some reference types (2, 3, 0xb) the value is used directly. Otherwise, the value is an index reference, which is either an ID2 value, or an offset, to be right shifted by 4 bits and used to fetch a pair from the index table to find the offset and size of the item in this descriptor block. The following reference types are known, but not all of these are implemented in the code yet. The following item types are known, but not all of these are implemented in the code yet. Associated Descriptor Item 0x7cec This style of descriptor block is similar to the 0xbcec format. This descriptor is also eventually decoded to a list of MAPI elements. Note the signature of 0x7cec. There are other descriptor block formats with other signatures. Note the indexOffset of 0x017a - starting at that position in the descriptor block, we have an array of two byte integers. The first integer (0x0006) is a (count-1) of the number of overlapping pairs following the count. The first pair is (0, 0xc), the next pair is (0xc, 0x14) and the last (7th) pair is (0x160, 0x179). These pairs are (start,end+1) offsets of items in this block. So we have count+2 integers following the count value. Note the 7coffset of 0x0040, which is an index reference. In this case, it is an internal reference pointer, which needs to be right shifted by 4 bits to become 0x0004, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0x14, 0xea) pair. We have the offset and size of the "7c" block located at offset 0x14 with a size of 214 bytes in this case. The "7c" block starts with a header with the following format: Note the b5Offset of 0x0020, which is an index reference. In this case, it is an internal reference pointer, which needs to be right shifted by 4 bits to become 0x0002, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xc, 0x14) pair. Finally, we have the offset and size of the "b5" block located at offset 0xc with a size of 8 bytes in this descriptor block. The "b5" block has the following format: Note the descoffset of 0x0060, which again is an index reference. In this case, it is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0006, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xea, 0xf0) pair. The datasize (2) plus the b5 code (04) gives the size of the entries, in this case 6 bytes. We now have the offset 0xea of an unused block of data in an unknown format, composed of 6 byte entries. That gives us (0xf0 - 0xea)/6 = 1, so we have a recordCount of one. We have seen cases where the descoffset in the b5 block is zero, and the index2Offset in the 7c block is zero. This has been seen for objects that seem to be attachments on messages that have been read. Before the message was read, it did not have any attachments. Note the index2Offset above of 0x0080, which again is an index reference. In this case, it is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0008, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xf0, 0x155) pair. This is an array of tables of four byte integers. We will call these the IND2 tables. The size of each of these tables is specified by the recordSize field of the "7c" header. The number of these tables is the above recordCount value derived from the "b5" block. Now the remaining data in the "7c" block after the header starts at offset 0x2a. There should be itemCount 8 byte items here, with the following format: The ind2Offset is a byte offset into the current IND2 table of some value. If that is a four byte integer value, then once we fetch that, we have the same triple (item type, reference type, value) as we find in the 0xbcec style descriptor blocks. If not, then this value is used directly. These 8 byte descriptors are processed recordCount times, each time using the next IND2 table. The item and reference types are as described above for the 0xbcec format descriptor block. 32 bit Associated Descriptor Item 0x0101 This descriptor block contains a list of I_ID values. It is used when an I_ID (that would normally point to a type 0x7cec or 0xbcec descriptor block) contains more data than can fit in any single descriptor of those types. In this case, it points to a type 0x0101 block, which contains a list of I_ID values that themselves point to the actual descriptor blocks. The total length value in the 0x0101 header is the sum of the lengths of the blocks pointed to by the list of I_ID values. The result is an array of subblocks, that may contain index references where the high order 16 bits specify which descriptor subblock to use. Only the first descriptor subblock contains the signature (0xbcec or 0x7cec). 64 bit Associated Descriptor Item 0x0101 This descriptor block contains a list of I_ID values, similar to the 32 bit version described above.