On 06 Feb 2020 at 09:49a, Avon pondered and said...

following bases FSX_MAG, FSX_HAM (but just spotted two messages with corruption in them, humm) FSX_ENG, and FSX_DAT also seem fine for order

The corrupted messages are a red herring. I found them also in the 1/100 base so have removed them... so they did export (albeit as corrupted) correctly.

I'm still trying to figure out why some bases when I rinse and repeat a fresh rescan are importing in non chronological order.

Will keep playing.

--- Mystic BBS v1.12 A44 2020/02/04 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Congrats and thanks for releasing this version :)

Agreed thanks for all your work on Mystic!

--- Mystic BBS v1.12 A44 2020/02/04 (Linux/64)
* Origin: Wicked BBS|wickedbbs.com:2333 (21:4/119.1)

Thanks. Hopefully it will be a smooth release. I don't think many people were using the later builds of the prealphas.

I updated most days. Last one prior to this that I used was from 2/2 I think. No problems so far.

Good to hear! I was cranking them out so fast that it would have been difficult for anyone to keep up!

I'm going to let A44 sit a few more days before I start working on A45 stuff, just to make sure there aren't any major issues that seem to pop up.

--- Mystic BBS v1.12 A44 2020/02/04 (Linux/64)
* Origin: Sector 7 (21:1/108)

following bases FSX_MAG, FSX_HAM (but just spotted two messages with corruption in them, humm) FSX_ENG, and FSX_DAT also seem fine for ord

The corrupted messages are a red herring. I found them also in the 1/100 base so have removed them... so they did export (albeit as corrupted) correctly.

Yep, unfortunately it can't fix the mess a previous A43 rescan made :(

I'm still trying to figure out why some bases when I rinse and repeat a fresh rescan are importing in non chronological order.

Will keep playing.

What are you max packet settings for size? If I understood correctly you're doing a huge 150,000 message rescan. I am wondering if its filling up packets and then running out of "extensions" to use for new packets so messages sort of get "wrapped" into other existing packets.

I'll think through this more once I can apply your settings into the logic.

--- Mystic BBS v1.12 A44 2020/02/04 (Linux/64)
* Origin: Sector 7 (21:1/108)

On 05 Feb 2020 at 05:31p, Greg Youngblood pondered and said...

Congrats and thanks for releasing this version :)

Agreed thanks for all your work on Mystic!

Not sure if that thanks was for me or g00r00 but if it was for me thanks :)

--- Mystic BBS v1.12 A44 2020/02/04 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

On 06 Feb 2020 at 01:26p, g00r00 pondered and said...

What are you max packet settings for size? If I understood correctly you're doing a huge 150,000 message rescan. I am wondering if its
filling up packets and then running out of "extensions" to use for new packets so messages sort of get "wrapped" into other existing packets. I'll think through this more once I can apply your settings into the logic.

The settings at 1/100 for /995 are

Address ³ 21:1/995
Domain ³ fsxnet
Session Type ³ BinkP
Archive Type ³ ZIP
Export Type ³ Hold
Route Info ³ 21:1/195.*
Max PKT Size ³ 512
Max ARC Size ³ 2048
Use Filebox ³ Hold
Filebox Dir ³ c:\mystfsx\filebox\fsxnet_z21n1n995\

Yes I am sending 1/100 a %rescan d=9999 areafix to it and getting back 36 compressed files with a ton of packets in them. I kept a copy of them so I could reset bases or delete bases at /995 to rerun the tests a few times.
I've only had intermittent BBSing time in the last few days due to family commitments but the retossing has been inconsistent. Some bases when I
retossed again did seem to land messages in order but other bases were
jumbled.

I am suspicious of the HUB bases where (so far) I have found issues with corrupt messages in two of the HUB bases and removed and packed the bases. I have yet to pull a full rescan of the HUB since doing this work. I wondered
if those corruptions may be a contributing factor to the issue I am trying to confirm/repro. But for now it;s a time thing for me to spend enough time on this to progress is :)

--- Mystic BBS v1.12 A44 2020/02/04 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Yes I am sending 1/100 a %rescan d=9999 areafix to it and getting back 36 compressed files with a ton of packets in them. I kept a copy of them so

This sort of points to maybe what I was saying before then. There are only 36 possible bundles so what happens if you rescan so many messages that all 36 bundles are used and at your maximums. I think at that point things may be hit or miss as to where they end up when they are exported.

Packets within a bundle are sorted and processed in date/time order but if there are packets that are spread out within many bundles, those obviously
are not - because bundles are processed only one at a time.

--- Mystic BBS v1.12 A44 2020/02/04 (Linux/64)
* Origin: Sector 7 (21:1/108)

On 07 Feb 2020 at 03:23p, g00r00 pondered and said...

Yes I am sending 1/100 a %rescan d=9999 areafix to it and getting bac compressed files with a ton of packets in them. I kept a copy of them

This sort of points to maybe what I was saying before then. There are only 36 possible bundles so what happens if you rescan so many messages that all 36 bundles are used and at your maximums. I think at that
point things may be hit or miss as to where they end up when they are exported.

Packets within a bundle are sorted and processed in date/time order but
if there are packets that are spread out within many bundles, those obviously are not - because bundles are processed only one at a time.

Do you think then if we're limited to 36 compressed packets we need to
increase the size of each compressed archive when a full rescan of a HUB such as d=9999 is called for by an areafix? It will lower the need to use all 36
if more packets can be stuffed inside a single compressed archive. Or do we need another way to send packets when everything is wanted?

I'm just trying to think how best to allow for a full rescan of a HUB
(assuming it's permitted and not overridden by the default_rescan = R=250 setting in mutil.ini.

Maybe Mystic should dynamically adjust the size of the compressed packets for a situation like this? Or perhaps there are additional settings in mutil.ini in the [ImportEchoMail] stanza to allow for differing max packet or max arc sizes and those are defined dependent upon the R or D values

Something like this perhaps?

msg_count <3000 | arc_size=2048
msg_count >3001 | arc_size=5120

day_count <1000 | arc_size=2048
day_count >1001 | arc_size=5120

I'm going to play some more and report back if I come up with anything
further using testing :)

--- Mystic BBS v1.12 A44 2020/02/04 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)