Re: FSAUS.PM\data\db folder is too big

Hello i'm using PM 13.11.84108 on windows 2008 r2 64 bit

my folder C:\Program Files (x86)\F-Secure\FSAUS.PM\data\db is about 46.5 Gb

it contains 4 folders: 7119644, 7119645, 7119647, 7119648

the folder 7119648 is the more recent and contains most of the data (about 46.3 Gb)

i checked the server.cfg file

daily_maintenance_time = "00:30","01:30","02:30","03:30","04:30","05:30","06:30","07:30","08:30","09:30","10:30","11:30","12:30","13:30","14:30","15:30","16:30","17:30","18:30","19:30","20:30","21:30","22:30","23:30"

last part of log files report:

Aug-01-15:30:08.961 4-INF 00000000 1944 Starting expiration of presentations
Aug-01-15:30:08.961 4-INF 00000000 1944 Finished expiration (0)
Aug-01-15:30:08.961 4-INF 00000000 1944 Beginning deletion of obsolete expired infopaks
Aug-01-15:30:08.961 4-INF 00000000 1944 Cleaning expired exposure group
Aug-01-15:30:08.961 4-INF 00000000 1944 Deleted 0 expired presentations
Aug-01-15:30:08.961 4-INF 00000000 1944 Starting deletion of old personal presentations
Aug-01-15:30:08.961 4-INF 00000000 1944 Finished deletion of old personal presentations
Aug-01-15:30:08.961 4-INF 00000000 1944 Starting deletion of presentations to be deleted
Aug-01-15:30:08.961 4-INF 00000000 1944 Finished deletion of presentations to be deleted (0)
Aug-01-15:30:08.961 4-INF 00000000 1944 Starting deletion of empty series
Aug-01-15:30:08.961 4-INF 00000000 1944 Finished deletion of empty series
Aug-01-15:30:08.961 4-INF 00000000 1944 Starting deletion of unused internal resources
Aug-01-15:30:08.961 4-INF 00000000 1944 Finished deletion of unused internal resources (0 resources were deleted)
Aug-01-15:30:08.961 4-INF 00000000 1944 Starting cleanup of obsolete import directories
Aug-01-15:30:08.961 4-INF 00000000 1944 Deleted 0 obsolete import directories
Aug-01-15:30:08.961 4-INF 00000000 1944 Channel directory daily backup was archived in C:\Program Files (x86)\F-Secure\FSAUS.PM/data/misc/backup/chandir_2018_08_01.bak
Aug-01-15:30:08.977 4-INF 00000000 1944 Starting cleanup of obsolete updatelog files
Aug-01-15:30:08.977 4-INF 00000000 1944 Finished cleanup of obsolete updatelog files

any hint to reduce space? thank you

Roberto

Find more posts tagged with

Comments

A_Grinkevitch

Thank you Roberto,

Installation packages (jars) indeed might utilize some extra space in the DB, but do not affect AUS diff handling. So, I guess host or PM services restart were the triggers to fix the situation... Would be interesting to figure out what exactly if the issue happens again... If PM services restart helps, probably AUS locks some files and unable to cleanup own directories (unlikely as in this case we might get hundreds of complains). If host restart resolves the issue, that might be related to file system.

Alexander

RobertoG

I think the action that actually started the db cleanup was when i delete old updates / version from then PM setup repository (intallation packages .JAR): I kept only the latest version of each release

during my tests I also restarted server and PM but dont think that was involved in the cleanup

the BIG db was started when I have already installed the latest version of PM so it wasnt a problem of updates

now server has 35 Gb free (when I noticed the problem it has only 1.7 gb)

MJ-perComp

very simple: "Patience"

A_Grinkevitch

Thank you for the update. Do you have any ideas what was the action bringing cleanup back to normal? Did you restart the host or PMS service?

Alexander

RobertoG

i will keep monitored the space in next days if new update will clean old messed ones. thanks you for hints.
Roberto

RobertoG

just a little update: the C:\Program Files (x86)\F-Secure\FSAUS.PM\data\db\7119648 folder is now 3 Gb less, but i havent done others configurations, so it seems that PM is auto-cleaning progressively...

A_Grinkevitch

Thank you Roberto,

Usual FSAUS.PM\data\db size is about 3 GB and might reach 5-6 GB between new publishing and cleanup. So, having it so huge is abnormal…

Folder “38”: Gemini update is pretty small, it has just 8 diffs 2K the largest. What are the files consuming so much disk space in that subfolder?

Folder “36”: First 10 diffs are ok, up to 5 MB, but numbers 11-35, diffs are about 250 MB each. Full Aquarius update is about 270 MB, so 90% of the update appeared as a diff. If everything goes smooth for next 25 versions, old (broken?) versions will go away and directory size will be back to normal…

In any way, I’d recommend to reinstall PM to clean all the FSAUS.PM data. Notice that it might bring some local traffic overhead as all 12 and older clients will redownload full updates.

Alexander

RobertoG

tnx for quick feedback

i remove old update (file .jar) from the server keeping only one per version (monitor show that i still have pc with client security version 9) i have some old win xp pc that seems cant upgraded. anyway i removede old update and se size decreased of about 15gb

my server.cfg parameter: num_old_versions_to_compare = 35

probably i will expand drive C of 10/20 gb (the server is a VM) to be safe and keep monitored the harddisk space

no i dont have restore backup

yes in pc list i have some computer long time offline (maybe some are dismissed pc no more used)

in my C:\Program Files (x86)\F-Secure\FSAUS.PM\data\db\7119648 dir I have 9 subfolder, all with date 1 or 2 august, in details:

folder "17" about 27mb

folder "18" about 24mb

folder "35" about 4,45gb

folder "36" about 6,73gb

folder "38" about 7,46gb

folder "39" about 6,50gb

folder "40" about 4,23gb

folder "41" about 6,26gb

folder "42" about 4gb

total 32.2 gb

content of info.iad of folder "38" is:

[presentation]
name="F-Secure Gemini Update 2018-08-02_02"
method="DataPak"
id=711964838
prs_category_id=0
installation_time="2018/08/02 10:11:30"
cookie="8xsMXlhOK04="
[versioning]
series_name="gemdb"
update_method="replace_previous"
version_number=711964887
version_seed=711964879
[publisher]
comment="1533193899"
[files]
1="fsgemu.db",2374182,70678329,"T19BGgZtkfga4bprPkUHzQ==",1564697
2="fsgemu.dll",2826720,1455239426,"+JYIHlWUN0AWpwPOAbjyOQ==",998287
3="gemini-update.ini",235,623249004,"gk+EtB2fFOEtDolJ2UVUQw==",197
4="gemini-update.mf",10116,300381158,"0Mk6HTkJ3aKKYDLZW6vAYQ==",4835
compression_level=6
compressed="infopak.zip",2568274,982805233,"qM3PsCFvpsTM68+mRihxqQ==",0
[location]
sub_channel="DB Updates"
group="Main"
[diffs]
from_prs_id=711964837
[diff_lists]
1=711964837,"_bwreserved_gdiff/_bwfilediffs.id711964837.zip",131,1788856066,"xo1KYaXAK8lQbMkYeEzF3g==","_bwreserved_gdiff/_bwfilediffszip.id711964837.zip",22,"ds2yutlYLSPB9vTYaCGNbA=="
2=711964832,"_bwreserved_gdiff/_bwfilediffs.id711964832.zip",456,1454539261,"DkK1PV1a7Is7UulBkYi6hA==","_bwreserved_gdiff/_bwfilediffszip.id711964832.zip",1129,"8CrjJ9LdsX63of22HCnqxA=="
3=711964831,"_bwreserved_gdiff/_bwfilediffs.id711964831.zip",457,1021062339,"yUKACF0NMGa6Qe7rriy9Sw==","_bwreserved_gdiff/_bwfilediffszip.id711964831.zip",1129,"8CrjJ9LdsX63of22HCnqxA=="
4=711964502,"_bwreserved_gdiff/_bwfilediffs.id711964502.zip",454,803491238,"uehXv35Tk8uzxNGP7svhrw==","_bwreserved_gdiff/_bwfilediffszip.id711964502.zip",1837,"iq+V+svsXdwpQU7ld6SH0w=="
5=711964501,"_bwreserved_gdiff/_bwfilediffs.id711964501.zip",454,983235781,"RIVqU6yVy0ouukK+uTA/1g==","_bwreserved_gdiff/_bwfilediffszip.id711964501.zip",1837,"iq+V+svsXdwpQU7ld6SH0w=="
6=711964465,"_bwreserved_gdiff/_bwfilediffs.id711964465.zip",454,1765173014,"mvDgGcEiZ5JmWuC5v4mkFA==","_bwreserved_gdiff/_bwfilediffszip.id711964465.zip",1953,"uVJo2bjKJ9RVf8Bn1y6k+w=="
7=711964462,"_bwreserved_gdiff/_bwfilediffs.id711964462.zip",454,2000657456,"wcbx4TR0Gf1acl23oyj64g==","_bwreserved_gdiff/_bwfilediffszip.id711964462.zip",1953,"uVJo2bjKJ9RVf8Bn1y6k+w=="
8=711964418,"_bwreserved_gdiff/_bwfilediffs.id711964418.zip",455,1519884674,"PxnVecuYa7HfZ8Ptwa5inQ==","_bwreserved_gdiff/_bwfilediffszip.id711964418.zip",2040,"5ZBeSvgjYwIJtjNf7rJdEw=="
; Do not delete or change this line __S

content of info.iad of folder "36" is:

[presentation]
name="F-Secure Aquarius Update 2018-08-02_01"
method="DataPak"
id=711964836
prs_category_id=0
installation_time="2018/08/02 08:55:01"
cookie="Gw1DZnEsON4="
[versioning]
series_name="aqualnx32"
update_method="replace_previous"
version_number=711965160
version_seed=711965006
[publisher]
comment="1533186086"
[files]
1="7zip.xmd",70464,303943415,"2y0IxELD/lOpIorKT5tB5g==",35668
2="access.xmd",6656,580388827,"D/MPSOmb3A58qCYlmr8AJg==",3995
3="ace.xmd",23456,1573217849,"TlovtzMQPjfx1wNSj/XfNg==",12498
4="adsntfs.xmd",9568,1558061200,"3fxtg8bQYca3YQDMfRwQ0Q==",5171
5="aitok.cvd",61092,1531372103,"qkbt1T4Sd226SBqkF3H3KA==",31361

....cut to limit in message size.....

941="yishield.xmd",2752,1233717791,"dkf46yg8DQPTSIVUNMKwfQ==",1541
942="z.xmd",2528,1452870970,"hpD7xuAplYuSz5KgI4u0KA==",1428
943="zip.xmd",111040,134055436,"ZWxrVw41o1ZxwhepO2aPqg==",54529
944="zoo.xmd",704,969059967,"0u22bEK2GvphoGbFt9pt1Q==",346
compression_level=6
compressed="infopak.zip",278607158,1565659200,"WGJTmR8G+zDSOvFcZA0T6w==",0
list="_bwfiles.txt",61809,225618712,"3+WvcYMFDitMJFaiIRrrOg==",0
compressed_list="_bwfiles.zip",32087,1110560907,"UZB1qmxumipF4HwyT/hGvA==",0
[location]
group="Expired"
initial_sub_channel="Special"
[diffs]
from_prs_id=711964833
[diff_lists]
1=711964833,"_bwreserved_gdiff/_bwfilediffs.id711964833.zip",131,1788856066,"xo1KYaXAK8lQbMkYeEzF3g==","_bwreserved_gdiff/_bwfilediffszip.id711964833.zip",22,"ds2yutlYLSPB9vTYaCGNbA=="
2=711964830,"_bwreserved_gdiff/_bwfilediffs.id711964830.zip",2598,1369170978,"K878w4pkHdVn4xZxCgLU2A==","_bwreserved_gdiff/_bwfilediffszip.id711964830.zip",2006318,"NdDoSc+5BcfIudVEwhcTRw=="
3=711964827,"_bwreserved_gdiff/_bwfilediffs.id711964827.zip",2598,746806447,"vXcu3kDlUHZH+D/dKcbcpw==","_bwreserved_gdiff/_bwfilediffszip.id711964827.zip",2006318,"NdDoSc+5BcfIudVEwhcTRw=="
4=711964825,"_bwreserved_gdiff/_bwfilediffs.id711964825.zip",3901,167188123,"KZqGe5uYKY2XRkXakzA8ew==","_bwreserved_gdiff/_bwfilediffszip.id711964825.zip",3143252,"broXTjHiyqM3y+IDFD8HZg=="
5=711964824,"_bwreserved_gdiff/_bwfilediffs.id711964824.zip",3901,1560810978,"8LJX91d9cr4hNP7Daq9KhA==","_bwreserved_gdiff/_bwfilediffszip.id711964824.zip",3143252,"broXTjHiyqM3y+IDFD8HZg=="
6=711964821,"_bwreserved_gdiff/_bwfilediffs.id711964821.zip",4818,1160054151,"/zcfz8WHeE6QOK+bQPjftA==","_bwreserved_gdiff/_bwfilediffszip.id711964821.zip",3163659,"UP3OTxEW4vwkDFdKkY1Ypg=="
7=711964819,"_bwreserved_gdiff/_bwfilediffs.id711964819.zip",4819,1694783874,"6lOhJGLQunh8DdrLi6p4Rw==","_bwreserved_gdiff/_bwfilediffszip.id711964819.zip",3163659,"UP3OTxEW4vwkDFdKkY1Ypg=="
8=711964814,"_bwreserved_gdiff/_bwfilediffs.id711964814.zip",5386,1241146364,"IfhhwQuoEDKE91vVuXR9SQ==","_bwreserved_gdiff/_bwfilediffszip.id711964814.zip",4107052,"tLox4+VCKo0m/dN9Ix5W3Q=="
9=711964812,"_bwreserved_gdiff/_bwfilediffs.id711964812.zip",5386,186769625,"spfRkAL5Rs4ufusrjb49bA==","_bwreserved_gdiff/_bwfilediffszip.id711964812.zip",4107052,"tLox4+VCKo0m/dN9Ix5W3Q=="
10=711964810,"_bwreserved_gdiff/_bwfilediffs.id711964810.zip",6259,677575664,"bunFMluu/LUQEDvxSmWeqA==","_bwreserved_gdiff/_bwfilediffszip.id711964810.zip",5296322,"cRbHbTI/jPJUGZc64R1N9g=="
11=711964809,"_bwreserved_gdiff/_bwfilediffs.id711964809.zip",32530,921218793,"ysxPXBeV7scU5qERpHpF4A==","_bwreserved_gdiff/_bwfilediffszip.id711964809.zip",254923284,"SWdHgYMZXYYJjagRlbvavw=="
12=711964801,"_bwreserved_gdiff/_bwfilediffs.id711964801.zip",32526,1286477955,"Q6YZgo09lku17/nURCjK0A==","_bwreserved_gdiff/_bwfilediffszip.id711964801.zip",254923291,"iqcZQT2EIezZC0TDRN997g=="
13=711964800,"_bwreserved_gdiff/_bwfilediffs.id711964800.zip",32526,340253191,"cw/wmBDwXRrhrvWRAyeUig==","_bwreserved_gdiff/_bwfilediffszip.id711964800.zip",254923291,"iqcZQT2EIezZC0TDRN997g=="
14=711964797,"_bwreserved_gdiff/_bwfilediffs.id711964797.zip",32527,1287266359,"n61pVnLuyHC5afdtaeNcVg==","_bwreserved_gdiff/_bwfilediffszip.id711964797.zip",254939304,"ZhmqRsFd3AtRFqiNEaSLBA=="
15=711964793,"_bwreserved_gdiff/_bwfilediffs.id711964793.zip",32527,1176263637,"9KSSKvoG841a3daojiaD/Q==","_bwreserved_gdiff/_bwfilediffszip.id711964793.zip",254939304,"ZhmqRsFd3AtRFqiNEaSLBA=="
16=711964790,"_bwreserved_gdiff/_bwfilediffs.id711964790.zip",33090,651306521,"h9l0uC9ZN0hR4PIlnrrKAQ==","_bwreserved_gdiff/_bwfilediffszip.id711964790.zip",254981630,"PNXrqa9bXPhIhwZcqyS+6Q=="
17=711964784,"_bwreserved_gdiff/_bwfilediffs.id711964784.zip",33168,311870345,"AvsmU635JLF5uyt3lOllJg==","_bwreserved_gdiff/_bwfilediffszip.id711964784.zip",254987172,"xPwKKiM9SfHuPRxBj+Xa1w=="
18=711964782,"_bwreserved_gdiff/_bwfilediffs.id711964782.zip",33168,991610726,"QDhcJr1Qf4Am7B9k1xjHAA==","_bwreserved_gdiff/_bwfilediffszip.id711964782.zip",254987172,"xPwKKiM9SfHuPRxBj+Xa1w=="
19=711964780,"_bwreserved_gdiff/_bwfilediffs.id711964780.zip",33241,434196240,"LtvrYkpEOCg8BzxEz06png==","_bwreserved_gdiff/_bwfilediffszip.id711964780.zip",254987516,"UMbYLqeEyxuqwkc+csq1LA=="
20=711964777,"_bwreserved_gdiff/_bwfilediffs.id711964777.zip",33402,1134421609,"uiiia9x60aQknxp9g6642Q==","_bwreserved_gdiff/_bwfilediffszip.id711964777.zip",254998750,"3gDVL4XY/L/ndKS0QrO1Bw=="
21=711964776,"_bwreserved_gdiff/_bwfilediffs.id711964776.zip",33402,137734937,"/y7ZExHPYwR2RZOQzYCAvg==","_bwreserved_gdiff/_bwfilediffszip.id711964776.zip",254999647,"EG5bnrpl1/NtPmKqsHc+Ag=="
22=711964773,"_bwreserved_gdiff/_bwfilediffs.id711964773.zip",33402,277507511,"AOfbXK+xgB/Z1eZN0cWbMQ==","_bwreserved_gdiff/_bwfilediffszip.id711964773.zip",254999647,"EG5bnrpl1/NtPmKqsHc+Ag=="
23=711964772,"_bwreserved_gdiff/_bwfilediffs.id711964772.zip",33729,829872469,"H9a0qwY+zg3q/Uuds7haIQ==","_bwreserved_gdiff/_bwfilediffszip.id711964772.zip",255009720,"6W99FjroexemzRgg7Z31zw=="
24=711964769,"_bwreserved_gdiff/_bwfilediffs.id711964769.zip",33729,1079074952,"ENMRk4WCFkMW9N25UlBvow==","_bwreserved_gdiff/_bwfilediffszip.id711964769.zip",255009720,"6W99FjroexemzRgg7Z31zw=="
25=711964767,"_bwreserved_gdiff/_bwfilediffs.id711964767.zip",33887,1236378839,"pkgugGihe06R/Bb1IYubAQ==","_bwreserved_gdiff/_bwfilediffszip.id711964767.zip",255040643,"+ihEVPseg/oelMb5LDhmLw=="
26=711964766,"_bwreserved_gdiff/_bwfilediffs.id711964766.zip",33886,1400539453,"y0lQVKdmDk7dnbyEyPWn0A==","_bwreserved_gdiff/_bwfilediffszip.id711964766.zip",255040643,"+ihEVPseg/oelMb5LDhmLw=="
27=711964763,"_bwreserved_gdiff/_bwfilediffs.id711964763.zip",33894,1915485799,"EOWsD7oj1EeNjW/glGFw+Q==","_bwreserved_gdiff/_bwfilediffszip.id711964763.zip",255042695,"UQ/X9t+cmjL5fyQXMdYAqA=="
28=711964761,"_bwreserved_gdiff/_bwfilediffs.id711964761.zip",33894,1854755054,"2xZzK7RrNsXLXJbmSLxDyg==","_bwreserved_gdiff/_bwfilediffszip.id711964761.zip",255042695,"UQ/X9t+cmjL5fyQXMdYAqA=="
29=711964760,"_bwreserved_gdiff/_bwfilediffs.id711964760.zip",33970,1182038638,"phUxiYhZEUKmfIJhMYiVbg==","_bwreserved_gdiff/_bwfilediffszip.id711964760.zip",255043993,"Svu8sug7Jz7HVhAv/mXjTQ=="
30=711964757,"_bwreserved_gdiff/_bwfilediffs.id711964757.zip",33970,1324729360,"24CrqfQ+71dK3/010wZUGw==","_bwreserved_gdiff/_bwfilediffszip.id711964757.zip",255043993,"Svu8sug7Jz7HVhAv/mXjTQ=="
31=711964751,"_bwreserved_gdiff/_bwfilediffs.id711964751.zip",34033,1372940028,"ICcCiMu9ebPV1QmeioAKrA==","_bwreserved_gdiff/_bwfilediffszip.id711964751.zip",255044569,"mvVLHk6BGcS7myneg3LGvw=="
32=711964748,"_bwreserved_gdiff/_bwfilediffs.id711964748.zip",34033,1327853880,"XKA+KRVjlD/4VfCuM1s8AQ==","_bwreserved_gdiff/_bwfilediffszip.id711964748.zip",255044569,"mvVLHk6BGcS7myneg3LGvw=="
33=711964746,"_bwreserved_gdiff/_bwfilediffs.id711964746.zip",34033,1252604201,"7TS4SYffxiFanVw+9HetvA==","_bwreserved_gdiff/_bwfilediffszip.id711964746.zip",255044610,"TEgdfeH/Lcg9xA9wV0HlOA=="
34=711964744,"_bwreserved_gdiff/_bwfilediffs.id711964744.zip",34033,864836517,"K6CuocGWT9TDgjcDqpUnAg==","_bwreserved_gdiff/_bwfilediffszip.id711964744.zip",255044610,"TEgdfeH/Lcg9xA9wV0HlOA=="
35=711964741,"_bwreserved_gdiff/_bwfilediffs.id711964741.zip",34104,2106403554,"MK6jx+A6BiDt59hp9gwkoA==","_bwreserved_gdiff/_bwfilediffszip.id711964741.zip",255045102,"/EDlxoZYqlVn3FSNza3f5Q=="
; Do not delete or change this line _R

both folder are dated august 2, 2018

thanks. Roberto

A_Grinkevitch

Hello Roberto,

Did you restore content in data\db folder from backup some time ago or might it be so that host was offline for some significant time? If so, diffs from old update version might be pretty big. Policy Manager reinstallation will clean AUS DB content properly.

If that is not the case, let's check for the reason... What is the value of num_old_versions_to_compare in C:\Program Files (x86)\F-Secure\FSAUS.PM\conf\server.cfg? If not default 35, it might explain so huge size.

Next, let's check the diff size:

Go to C:\Program Files (x86)\F-Secure\FSAUS.PM\data\db\7119648 and check which subdirectories have size exceeding 1 GB. There should be just 2 of those, or max 4 if new updates were published recently. 2 extra dirs will be cleaned in half an hour according to daily_maintenance_time schedule.

Please open info.iad files from two latest huge subfolders and copy-paste [diff_lists] section content here.

TIA,

Alexander

MJ-perComp

F-Secure Automatic Update Server (AUS) can be a pretty space consuming module. Everything depends on how many updates F-Secure has to publish in a specific timerange.
The more often F-secure has to push an update the more versions need to be downloaded to cover 72h. The limit is currently set 35 versions. These are differentials to the previous versions.
From time to time F-Secure has to publish cleanups for the databases. As they are a kind of wipe-through the differentials are pretty big and sum up to numbers you see.

So:
1) This is expected
2) You can use FSAUARESET to wipe all data once, but a) means a lot of traffic from PMS to Clients and b) the issue will return unless you grant more sapce (see below).

3) As Alex already wrote GUTS will resolve the issue, but requires all hosts to be V13+

Easy workaround:
Stop FSAUS
rename the Folder c:\Program Files (x86)\F-Secure\FSAUS.PM\data\
to ...data.old
Create a new folder ...\data
Create a new Partiton with some 50 GB BUT

do NOT assign a driveletter instead mount it to the above created data folder

Move all content from data.old to data (this takes some time as it involves a different partition).
delete data.old

restart FSAUS.

From Win2012 you could easily expand the system drive, the benefit of above is, that AUS will no longer write to your system-partition.

An other way would be to change all path inside AUS-Configuration, but as F-Secure overwrites that pretty regularly ignoring the changes (just doing a backup) the problem returns.