diff --git a/NEWS b/NEWS index 6f44ba7..934e6fa 100644 --- a/NEWS +++ b/NEWS @@ -1,14 +1,17 @@ - $Id$ - +0.6.6 2008-01-31 Code cleanup, switch from cvs to mercurial source control. +0.6.5 2008-01-22 Code cleanup, rpm group Applications/Productivity. +0.6.4 2008-01-19 More fixes for 64 bit format, merge changes from svn Alioth. +0.6.3 2008-01-13 More type consistency issues found by splint. +0.6.2 2008-01-12 More fixes for 64 bit format, consistent types size_t, off_t, etc. 0.6.1 2008-01-06 Outlook 2003 64 bit format and fix for bogus contacts. 0.5.12 2007-10-02 security fix for possible buffer overruns in liv-zemple decoding 0.5.11 2007-08-24 fix for unitialized variable 0.5.10 2007-08-20 fix yet more valgrind errors, restructure readpst recursive walk, backwards overrun test 0.5.9 2007-08-12 fix more valgrind errors, pst2ldif wrote undefined data 0.5.8 2007-08-10 lzfu_decompress/base64_encode encoded random data into attachment 0.5.7 2007-08-09 fix valgrind errors, using uninitialized data 0.5.6 2007-07-15 handle small pst files, better decoding of 7c blocks 0.5.5 2007-07-10 merge changes from Joe Nahmias version 0.5.4 2006-02-25 add MH mode, generated filenames with no leading zeros 0.5.3 2006-02-20 switch to gnu autoconf/automake 0.5.2 2006-02-18 add pst2ldif, fix btree processing in libpst.c diff --git a/xml/libpst.in b/xml/libpst.in index b9e2bc3..6b70416 100644 --- a/xml/libpst.in +++ b/xml/libpst.in @@ -1,1882 +1,1882 @@ @PACKAGE@ Utilities - Version @VERSION@ Packages This is a fork of the libpst project at SourceForge. Another fork is located at http://alioth.debian.org/projects/libpst/ The various source and binary packages are available at http://www.five-ten-sg.com/@PACKAGE@/packages/ The most recent documentation is available at http://www.five-ten-sg.com/@PACKAGE@/ - A Mercurial source code repository is available at http://hg.five-ten-sg.com/libpst. + A Mercurial source code repository for this project is available at http://hg.five-ten-sg.com/@PACKAGE@/. This version can now convert both 32 bit Outlook files (pre 2003), and the 64 bit Outlook 2003 pst files. 2008-01-27 readpst 1 readpst @VERSION@ readpst convert PST (MS Outlook Personal Folders) files to mbox and other formats Synopsis readpst pstfile Description readpst is a program that can read an Outlook PST (Personal Folders) file and convert it into an mbox file, a format suitable for KMail, a recursive mbox structure, or separate emails. Options -b Do not save the attachments for the RTF format of the email body. -C Decrypt the entire pst file and dump it to stdout. -c format Set the Contact output mode. Use -cv for vcard format or -cl for an email list. -d debug-file Specify name of debug log file. The log file is not an ascii file, it is a binary file readable by readpstlog. -h Show summary of options and exit. -k Changes the output format to KMail. -o output-directory Specifies the output directory. The directory must already exist, and is entered after the PST file is opened, but before any processing of files commences. -q Changes to silent mode. No feedback is printed to the screen, except for error messages. -r Changes the output format to Recursive. This will create folders as named in the PST file, and will put all emails in a file called "mbox" inside each folder. These files are then compatible with all mbox-compatible email clients. -S Output messages into separate files. This will create folders as named in the PST file, and will put each email in its own file. These files will be numbered from 1 increasing in intervals of 1 (ie 1, 2, 3, ...). Any attachments are saved alongside each email as XXXXXXXXX-attach1, XXXXXXXXX-attach2 and so on, or with the name of the attachment if one is present. -M Output messages in MH format as separate files. This will create folders as named in the PST file, and will put each email together with any attachments into its own file. These files will be numbered from 1 to n with no leading zeros. -V Show program version and exit. -w Overwrite any previous output files. Beware: When used with the -S switch, this will remove all files from the target folder before writing. This is to keep the count of emails and attachments correct. See Also readpstlog 1 Author This manual page was originally written by Dave Smith <dave.s@earthcorp.com>, and updated by Joe Nahmias <joe@nahmias.net> for the Debian GNU/Linux system (but may be used by others). It was subsequently updated by Brad Hards <bradh@frogmouth.net>, and converted to xml format by Carl Byington <carl@five-ten-sg.com>. Copyright Copyright (C) 2002 by David Smith <dave.s@earthcorp.com>. XML version Copyright (C) 2006 by 510 Software Group <carl@five-ten-sg.com>. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2008-01-27 lspst 1 lspst @VERSION@ lspst list PST (MS Outlook Personal Folders) file data Synopsis lspst pstfile Options -d debug-file Specify name of debug log file. The log file is not an ascii file, it is a binary file readable by readpstlog. -h Show summary of options and exit. -V Show program version and exit. Description lspst is a program that can read an Outlook PST (Personal Folders) file and produce a simple listing of the data (contacts, email subjects, etc). See Also readpstlog 1 Author lspst was written by Joe Nahmias <joe@nahmias.net> based on readpst. This man page was written by 510 Software Group <carl@five-ten-sg.com>. Copyright Copyright (C) 2004 by Joe Nahmias <joe@nahmias.net>. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2008-01-27 readpstlog 1 readpstlog @VERSION@ readpstlog convert a readpst logfile to text format Synopsis readpstlog logfile Description readpstlog is a program that converts the binary logfile generated by readpst to a more desirable text format. Options -f format Sets the format of the text log output. Currently, the only valid output formats are T, for single line text, D for the default default multi line format, and I for an indented style with single line text. -t include-types Print only the specified types of log messages. Types are specified in a comma-delimited list (e.g. 3,10,5,6). -x exclude-types Exclude the specified types of log messages. Types are specified in a comma-delimited list (e.g. 3,10,5,6). Message Types readpstlog understands the following types of log messages: 1 File accesses 2 Index accesses 3 New email found 4 Warnings 5 Read accesses 6 Informational messages 7 Main function calls 8 Decrypting calls 9 Function entries 10 Function exits 11 HexDump calls Author This manual page was written by Joe Nahmias <joe@nahmias.net> for the Debian GNU/Linux system (but may be used by others). It was converted to xml format by Carl Byington <carl@five-ten-sg.com>. Copyright Copyright (C) 2002 by David Smith <dave.s@earthcorp.com>. XML version Copyright (C) 2005 by 510 Software Group <carl@five-ten-sg.com>. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2008-01-27 pst2ldif 1 pst2ldif @VERSION@ pst2ldif extract contacts from a MS Outlook .pst file in .ldif format Synopsis pst2ldif pstfilename Options -h Show summary of options. Subsequent options are then ignored. -V include-types Show program version. Subsequent options are then ignored. -b ldap-base Sets the ldap base value used in the dn records. You probably want to use something like "o=organization, c=US". -c class Sets the objectClass values for the contact items. This class needs to be defined in the schema used by your LDAP server, and at a minimum it must contain the ldap attributes given below. -d debug-file Specify name of debug log file. The log file is not an ascii file, it is a binary file readable by readpstlog. Description pst2ldif reads the contact information from a MS Outlook .pst file and produces a .ldif file that may be used to import those contacts into an LDAP database. The following ldap attributes are generated: cn givenName sn personalTitle company mail postalAddress l st postalCode c homePhone telephoneNumber facsimileTelephoneNumber mobile description Copyright Copyright (C) 2006 by 510 Software Group <carl@five-ten-sg.com> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program; see the file COPYING. If not, please write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. Version @VERSION@ 2008-01-27 outlook.pst 5 outlook.pst format of MS Outlook .pst file Synopsis outlook.pst Overview Each item in a .pst file is identified by two id values ID1 and ID2. There are two separate b-trees indexed by these ID1 and ID2 values. Starting with Outlook 2003, the file format changed from one with 32 bit pointers, to one with 64 bit pointers. We describe both formats here. 32 bit File Header The 32 bit file header is located at offset 0 in the .pst file. We only support index types 0x0e and 0x17, and encryption types 0x00 and 0x01. Index type 0x0e is the older 32 bit Outlook format. Index type 0x17 is the newer 64 bit Outlook format. Encryption type 0x00 is no encryption, and type 0x01 is the only other supported encryption type. offsetIndex1 is the file offset of the root of the index1 b-tree, which contains (ID1, offset, size, unknown) tuples for each item in the file. backPointer1 is the value that should appear in the parent pointer of that root node. offsetIndex2 is the file offset of the root of the index2 b-tree, which contains (ID2, DESC-ID1, LIST-ID1, PARENT-ID2) tuples for each item in the file. backPointer2 is the value that should appear in the parent pointer of that root node. 64 bit File Header The 64 bit file header is located at offset 0 in the .pst file. 32 bit Index 1 Node The 32 bit index1 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 12 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (ID1, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and ID1 is the lowest ID1 value in the subtree. 64 bit Index 1 Node The 64 bit index1 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 24 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (ID1, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and ID1 is the lowest ID1 value in the subtree. 32 bit Index 1 Leaf Node The 32 bit index1 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 12 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (ID1, offset, size, unknown) The two low order bits of the ID1 value seem to be flags. I have never seen a case with bit zero set. Bit one indicates that the item is not encrypted. Note that references to these ID1 values elsewhere may have the low order bit set (and I don't know what that means), but when we do the search in this tree we need to clear that bit so that we can find the correct item. 64 bit Index 1 Leaf Node The 64 bit index1 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 24 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (ID1, offset, size, unknown) The two low order bits of the ID1 value seem to be flags. I have never seen a case with bit zero set. Bit one indicates that the item is not encrypted. Note that references to these ID1 values elsewhere may have the low order bit set (and I don't know what that means), but when we do the search in this tree we need to clear that bit so that we can find the correct item. 32 bit Index 2 Node The 32 bit index2 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 12 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (ID2, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and ID2 is the lowest ID2 value in the subtree. 64 bit Index 2 Node The 64 bit index2 b-tree nodes are 512 byte blocks with the following format. The itemCount specifies the number of 24 byte records that are active. The nodeLevel is non-zero for this style of nodes. The leaf nodes have a different format. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a triple of (ID2, backPointer, offset) where the offset points to the next deeper node in the tree, the backPointer value must match the backPointer in that deeper node, and ID2 is the lowest ID2 value in the subtree. 32 bit Index 2 Leaf Node The 32 bit index2 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 16 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (ID2, DESC-ID1, LIST-ID1, PARENT-ID2) 64 bit Index 2 Leaf Node The 64 bit index2 b-tree leaf nodes are 512 byte blocks with the following format. The itemCount specifies the number of 32 byte records that are active. The nodeLevel is zero for these leaf nodes. The backPointer must match the backPointer from the triple that pointed to this node. Each item in this node is a tuple of (ID2, DESC-ID1, LIST-ID1, PARENT-ID2) 32 bit Associated List Item 0x0002 Contains associations between id1 and id2 for the items controlled by the record. In the above 32 bit leaf node, we have a tuple of (0x61, 0x02a82c, 0x02a836, 0) 0x02a836 is the ID1 of the associated list, and we can lookup that ID1 value in the index1 b-tree to find the (offset,size) of the data in the .pst file. 64 bit Associated List Item 0x0002 Contains associations between id1 and id2 for the items controlled by the record. Associated Descriptor Item 0xbcec Contains information about the item, which may be email, contact, or other outlook types. In the above leaf node, we have a tuple of (0x21, 0x00e638, 0, 0) 0x00e638 is the ID1 of the associated descriptor, and we can lookup that ID1 value in the index1 b-tree to find the (offset,size) of the data in the .pst file. Note the signature of 0xbcec. There are other descriptor block formats with other signatures. Note the indexOffset of 0x013c - starting at that position in the descriptor block, we have an array of two byte integers. The first integer (0x000b) is a (count-1) of the number of overlapping pairs following the count. The first pair is (0, 0xc), the next pair is (0xc, 0x14) and the last (12th) pair is (0x123, 0x13b). These pairs are (start,end+1) offsets of items in this block. So we have count+2 integers following the count value. Note the b5offset of 0x0020, which is a type that I will call an index reference. Such index references have at least two different forms, and may point to data either in this block, or in some other block. External pointer references have the low order 4 bits all set, and are ID2 values that can be used to fetch data. This value of 0x0020 is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0002, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xc, 0x14) pair. So far we have only described internal index references where the high order 16 bits are zero. That suffices for single descriptor blocks. But in the case of the type 0x0101 descriptor block, we have an array of subblocks. In this case, the high order 16 bits of an internal index reference are used to select the subblock. Each subblock starts with a 16 bit indexOffset which points to the count and array of 16 bit integer pairs which are offsets in the current subblock. Finally, we have the offset and size of the "b5" block located at offset 0xc with a size of 8 bytes in this descriptor block. The "b5" block has the following format: Note the descoffset of 0x0040, which again is an index reference. In this case, it is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0004, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0x14, 0x7c) pair. We now have the offset 0x14 of the descriptor array, composed of 8 byte entries. Each descriptor entry has the following format: For some reference types (2, 3, 0xb) the value is used directly. Otherwise, the value is an index reference, which is either an ID2 value, or an offset, to be right shifted by 4 bits and used to fetch a pair from the index table to find the offset and size of the item in this descriptor block. The following reference types are known, but not all of these are implemented in the code yet. The following item types are known, but not all of these are implemented in the code yet. Associated Descriptor Item 0x7cec This style of descriptor block is similar to the 0xbcec format. Note the signature of 0x7cec. There are other descriptor block formats with other signatures. Note the indexOffset of 0x017a - starting at that position in the descriptor block, we have an array of two byte integers. The first integer (0x0006) is a (count-1) of the number of overlapping pairs following the count. The first pair is (0, 0xc), the next pair is (0xc, 0x14) and the last (7th) pair is (0x160, 0x179). These pairs are (start,end+1) offsets of items in this block. So we have count+2 integers following the count value. Note the 7coffset of 0x0040, which is an index reference. In this case, it is an internal reference pointer, which needs to be right shifted by 4 bits to become 0x0004, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0x14, 0xea) pair. We have the offset and size of the "7c" block located at offset 0x14 with a size of 214 bytes in this case. The "7c" block starts with a header with the following format: Note the b5Offset of 0x0020, which is an index reference. In this case, it is an internal reference pointer, which needs to be right shifted by 4 bits to become 0x0002, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xc, 0x14) pair. Finally, we have the offset and size of the "b5" block located at offset 0xc with a size of 8 bytes in this descriptor block. The "b5" block has the following format: Note the descoffset of 0x0060, which again is an index reference. In this case, it is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0006, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xea, 0xf0) pair. That gives us (0xf0 - 0xea)/6 = 1, so we have a recordCount of one. The actual data between 0xea and 0xf0 is unknown and unused here. Note the index2Offset above of 0x0080, which again is an index reference. In this case, it is an internal pointer reference, which needs to be right shifted by 4 bits to become 0x0008, which is then a byte offset to be added to the above indexOffset plus two (to skip the count), so it points to the (0xf0, 0x155) pair. This is an array of tables of four byte integers. We will call these the IND2 tables. The size of each of these tables is specified by the recordSize field of the "7c" header. The number of these tables is the above recordCount value derived from the "b5" block. Now the remaining data in the "7c" block after the header starts at offset 0x2a. There should be itemCount 8 byte items here, with the following format: The ind2Offset is a byte offset into the current IND2 table of some value. If that is a four byte integer value, then once we fetch that, we have the same triple (item type, reference type, value) as we find in the 0xbcec style descriptor blocks. If not, then this value is used directly. These 8 byte descriptors are processed recordCount times, each time using the next IND2 table. The item and reference types are as described above for the 0xbcec format descriptor block. 32 bit Associated Descriptor Item 0x0101 This descriptor block contains a list of ID1 values. It is used when an ID1 (that would normally point to a type 0x7cec or 0xbcec descriptor block) contains more data than can fit in any single descriptor of those types. In this case, it points to a type 0x0101 block, which contains a list of ID1 values that themselves point to the actual descriptor blocks. The total length value in the 0x0101 header is the sum of the lengths of the blocks pointed to by the list of ID1 values. The result is an array of subblocks, that may contain index references where the high order 16 bits specify which descriptor subblock to use. Only the first descriptor subblock contains the signature (0xbcec or 0x7cec). 64 bit Associated Descriptor Item 0x0101 This descriptor block contains a list of ID1 values.