Discussion:
CCKD or not, Shadow Files or not
(too old to reply)
brian_westerman
2009-03-18 09:35:27 UTC
Permalink
Hi,

I have been running for a few years now with my DASD in CCKD format and use Shadow files. In speaking with another user last night, it came up that he was not using CCKD because he felt it left the possibility to corrupt wide open. He did not use Shadow files because he didn't really trust them either.

I have never (until today) had any problems with CCKD (compressed DASD) and shadow files. But today I decided to "merge" one of my DASD volumes shadow file to the base. There was a "free space" issue, and while the sf- command said everything was fine, when I next IPLed the system, most of that volume was unusable. I was able to go back to my original BASE+SHADOW, and things are fine, but it made me wonder. Am I doing something inherently bad by using CCKD and Shadow files?

I was wondering how others on the list are set up.

Can you guys respond back with your thoughts on this?

Brian
Gerhard Postpischil
2009-03-18 15:05:08 UTC
Permalink
Post by brian_westerman
I have never (until today) had any problems with CCKD
(compressed DASD) and shadow files. But today I decided to
"merge" one of my DASD volumes shadow file to the base.
There was a "free space" issue, and while the sf- command
said everything was fine, when I next IPLed the system, most
of that volume was unusable. I was able to go back to my
original BASE+SHADOW, and things are fine, but it made me
wonder. Am I doing something inherently bad by using CCKD
and Shadow files?
My first Hercules/MVS (turnkey 3 with updates) was based on an
Iomega USB drive. My area has dirty power with frequent short
interruptions, and despite my UPS I kept losing either the
system or the disk. Almost every time I wound up with
cross-linked files, missing data, etc. This seemed to be worst
on the CCKD files (with or without shadows). Finally after one
outage the Iomega wasn't recognized by the system at all, and
wouldn't even accept a low-level format. It's now a paper weight.

I switched to Western Digital drives, and uncompressed disk
files. I use shadow files only for unwanted stuff, such as the
SORT packs, and (dedicated) work and SYSDA packs (and delete
them automatically in the Hercules batch file). Having
uncompressed files without shadow files also means I'll have no
nasty surprises running out of disk space trying to update a
disk. I take weekly backups under Windows, and MVS file backups
after critical updates. In the last two years I've lost about an
hours work or so, but that was due to an error in RPF Edit.




Gerhard Postpischil
Bradford, VT
BruceTSmith
2009-03-19 00:46:03 UTC
Permalink
Two things. First I had a noticeable performance decrease on CCKD, vs the exact system on CKD.

And second, with the size of PC drives these days, do we still need it? A few years ago when drives were much smaller it was a good idea. But today, I mean I could setup a whole string of 16 uncompressed 3390-3s and it wouldn't take even half of the free space. And that's just on my C: drive...

BTS :)))
Post by Gerhard Postpischil
Post by brian_westerman
I have never (until today) had any problems with CCKD
(compressed DASD) and shadow files. But today I decided to
"merge" one of my DASD volumes shadow file to the base.
There was a "free space" issue, and while the sf- command
said everything was fine, when I next IPLed the system, most
of that volume was unusable. I was able to go back to my
original BASE+SHADOW, and things are fine, but it made me
wonder. Am I doing something inherently bad by using CCKD
and Shadow files?
My first Hercules/MVS (turnkey 3 with updates) was based on an
Iomega USB drive. My area has dirty power with frequent short
interruptions, and despite my UPS I kept losing either the
system or the disk. Almost every time I wound up with
cross-linked files, missing data, etc. This seemed to be worst
on the CCKD files (with or without shadows). Finally after one
outage the Iomega wasn't recognized by the system at all, and
wouldn't even accept a low-level format. It's now a paper weight.
I switched to Western Digital drives, and uncompressed disk
files. I use shadow files only for unwanted stuff, such as the
SORT packs, and (dedicated) work and SYSDA packs (and delete
them automatically in the Hercules batch file). Having
uncompressed files without shadow files also means I'll have no
nasty surprises running out of disk space trying to update a
disk. I take weekly backups under Windows, and MVS file backups
after critical updates. In the last two years I've lost about an
hours work or so, but that was due to an error in RPF Edit.
Gerhard Postpischil
Bradford, VT
Tony Harminc
2009-03-19 16:37:40 UTC
Permalink
Post by BruceTSmith
Two things. First I had a noticeable performance decrease on CCKD, vs the exact system on CKD.
I think this depends greatly on your CPU speed, the number of (host)
processors, and the ratio of reads to writes.
Post by BruceTSmith
And second, with the size of PC drives these days, do we still need it? A few years ago when drives were much smaller it was a good idea. But today, I mean I could setup a whole string of 16 uncompressed 3390-3s and it wouldn't take even half of the free space. And that's just on my C: drive...
Saving space isn't the only reason for compressing (and I mean this
generally, and not just for CCKD). Particularly in a read-heavy
environment, where the decompressor typically runs quite fast, the
time saved by transferring less data across the I/O interface can make
things faster rather than slower. The compressor is typically a good
deal slower than the decompressor, so if you are writing heavily, it
may well slow things down. But even then, CCKD is sensitive to the
current performance situation, and will compress less or even not at
all if things are looking dicey CPU-wise.

Tony H.
Gunnar Opheim
2009-03-19 23:26:05 UTC
Permalink
Has anybody compared CCKD to CKD on Windows NTFS compressed files, in terms of
size or performance?

Gunnar


----- Original Message -----
From: "Tony Harminc" <tharminc-***@public.gmane.org>

Saving space isn't the only reason for compressing (and I mean this
generally, and not just for CCKD). Particularly in a read-heavy
environment, where the decompressor typically runs quite fast, the
time saved by transferring less data across the I/O interface can make
things faster rather than slower. The compressor is typically a good
deal slower than the decompressor, so if you are writing heavily, it
may well slow things down. But even then, CCKD is sensitive to the
current performance situation, and will compress less or even not at
all if things are looking dicey CPU-wise.

Tony H.
Mike Schwab
2009-03-20 03:28:14 UTC
Permalink
As far as size, the CKD is always the full size of the disk. A full
volume on CCKD uses about 25-30% of the size of the disk, empty and
erased / not used volumes much less.
Post by Gunnar Opheim
Has anybody compared CCKD to CKD on Windows NTFS compressed files, in terms of
size or performance?
Gunnar
--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?
brian_westerman
2009-03-21 04:05:35 UTC
Permalink
So it appears that some people like it compressed and some do not. In 8 years I have only had one problem (my SF- error last week), and that was easily recoverable since I backup the shadow files daily, and the BASE files are backed up about every 6 months (although they are READ only and never change, it never hurts to have extra copies).

I didn't loose any data, but I began to wonder if I had been advising people wrong or not over the years. It appears (from my point of view), that if you don't mind the extra space being used by non-compressed dasd and if you don't mind backing up a LOT of space with each backup, then there is not reason for you to HAVE to use compressed CCKD and SHADOW files. (I'm not sure why you would want to use compressed CCKD and NOT use shadow files, but I'm sure people would do it even if only to make backing up the volumes quicker).

Any way, I think that since my system have VERY few updates of the DASD files, except for maintenance and normal program editing, etc., that my choice of READONLY BASE CCKD with Shadow files is just as good as the other alternatives, so I'll stick with it.

I want to thank you all for your input. It was very informative.

Brian
Post by Mike Schwab
As far as size, the CKD is always the full size of the disk. A full
volume on CCKD uses about 25-30% of the size of the disk, empty and
erased / not used volumes much less.
Post by Gunnar Opheim
Has anybody compared CCKD to CKD on Windows NTFS compressed files, in terms of
size or performance?
Gunnar
--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?
halfmeg
2009-03-21 12:48:30 UTC
Permalink
<snip>
It appears (from my point of view), that if you don't mind the extra
space being used by non-compressed dasd and if you don't mind
backing up a LOT of space with each backup, ...
<snip>
but I'm sure people would do it even if only to make backing up the
volumes quicker).
<snip>
One way to reduce the backup of empty space and make backing up quicker would be to use CCKD with no compression. DASDCOPY has the -0 option to allow conversion between various types of dasd to CCKD no compression. One could execute something similiar to:

dasdcopy -0 -o CCKD mvsres.xxx mvsres.xxx.CCKD

in order to see how much space would be saved for backups.

One of the MVSRES packs available on the web went from 326,861,312 down to 31,226,244 using the above.

Phil
Gunnar Opheim
2009-03-21 18:23:51 UTC
Permalink
Post by Mike Schwab
As far as size, the CKD is always the full size of the disk. A full
volume on CCKD uses about 25-30% of the size of the disk, empty and
erased / not used volumes much less.
Post by Gunnar Opheim
Has anybody compared CCKD to CKD on Windows NTFS compressed files, in terms of
size or performance?
Gunnar
CKD, uncompressed, is of course full size. My question was if anybody had
compared NTFS compression to CCKD compression.

Gunnar

tsagoth
2009-03-20 15:24:33 UTC
Permalink
Is this known to be true, or is it just a belief ? With a good raid controller and sata drives, you can get a solid 200MB/sec transfer speed and even more as a burst. Given the size of a cylinder vs the transfer speed I've become skeptical that cckd yields any performance benefit. So to test it out I've started converting a set of drives to full size. Once that's done I'll do some transfer tests with some volumes and see how it turns out.
Post by Tony Harminc
Saving space isn't the only reason for compressing (and I mean this
generally, and not just for CCKD). Particularly in a read-heavy
environment, where the decompressor typically runs quite fast, the
time saved by transferring less data across the I/O interface can make
things faster rather than slower.
Tony H.
Tony Harminc
2009-03-20 15:42:47 UTC
Permalink
Post by tsagoth
Is this known to be true, or is it just a belief ? With a good raid controller and sata drives, you can get a solid 200MB/sec transfer speed and even more as a burst. Given the size of a cylinder vs the transfer speed I've become skeptical that cckd yields any performance benefit. So to test it out I've started converting a set of drives to full size. Once that's done I'll do some transfer tests with some volumes and see how it turns out.
Well, it's more than just a belief - it's well-known and has been
demonstrated and measured in the past. But as I said, I'm talking
generally, and whether it applies with any particular hardware in the
case of CCKD certainly needs to be measured. The principle will not be
invalidated if your measurements say it doesn't speed things up. But
the results will be interesting.

Tony H.
Post by tsagoth
Post by Tony Harminc
Saving space isn't the only reason for compressing (and I mean this
generally, and not just for CCKD). Particularly in a read-heavy
environment, where the decompressor typically runs quite fast, the
time saved by transferring less data across the I/O interface can make
things faster rather than slower.
Tony H.
Jay Maynard
2009-03-20 16:33:27 UTC
Permalink
Post by tsagoth
Is this known to be true, or is it just a belief ? With a good raid
controller and sata drives, you can get a solid 200MB/sec transfer speed
and even more as a burst. Given the size of a cylinder vs the transfer
speed I've become skeptical that cckd yields any performance benefit. So
to test it out I've started converting a set of drives to full size. Once
that's done I'll do some transfer tests with some volumes and see how it
turns out.
What's generally been found over the years is that CCKD is faster on
machines without hardware acceleration of disk I/O. The cost of the extended
I/O transfer swamps the cost of the uncompression.

If you've got hardware RAID, that equation may well change. Hardware RAID is
quite uncommon.
--
Jay Maynard, K5ZC http://www.conmicro.com
http://jmaynard.livejournal.com http://www.tronguy.net
http://www.hercules-390.org (Yes, that's me!)
Buy Hercules stuff at http://www.cafepress.com/hercules-390
somitcw
2009-03-21 01:09:39 UTC
Permalink
Post by tsagoth
Post by Tony Harminc
Saving space isn't the only reason for compressing
(and I mean this generally, and not just for CCKD).
Particularly in a read-heavy environment, where the
decompressor typically runs quite fast, the time
saved by transferring less data across the I/O
interface can make things faster rather than slower.
Tony H.
Is this known to be true, or is it just a belief ?
With a good raid controller and sata drives, you
can get a solid 200MB/sec transfer speed and even
more as a burst. Given the size of a cylinder vs
the transfer speed I've become skeptical that cckd
yields any performance benefit. So to test it out
I've started converting a set of drives to full
size. Once that's done I'll do some transfer tests
with some volumes and see how it turns out.
How about this test. I created two 3390-3
disk volues. One was -z -a and the other was
-a -lfs so one is compressed and one not.

CCKD.CCKD .,...,..5,766 bytes
CKD.CKD is 8,541,855,878 bytes

At your solid 200MB/sec transfer speed, for
a program reading all 150,300 tracks to check
for an EOF on each, which should transfer faster?
Only track 0 and maybe a dummy VTOC track would
have to decompress.

For another test, fill all 150,300 tracks with
data that will not compress. Both disks would
then be about the same size and transfer rate
about the same and no compression so no
decompression. i.e. Probably a wash.

Any less contrived test should show that CPU
decompression overhead is probably less than CPU
and clock time that would have been spent
transferring the larger amount of data.
g. desbiens
2009-03-19 02:47:06 UTC
Permalink
my 2¢... On occasion I would end up with a corrupt cckd image after merging with the sf- command. I now avoid it like the plague and use dasdcopy instead to merge shadow files into base. Never had a problem since.

gd
Post by brian_westerman
Hi,
I have been running for a few years now with my DASD in CCKD format and use Shadow files. In speaking with another user last night, it came up that he was not using CCKD because he felt it left the possibility to corrupt wide open. He did not use Shadow files because he didn't really trust them either.
I have never (until today) had any problems with CCKD (compressed DASD) and shadow files. But today I decided to "merge" one of my DASD volumes shadow file to the base. There was a "free space" issue, and while the sf- command said everything was fine, when I next IPLed the system, most of that volume was unusable. I was able to go back to my original BASE+SHADOW, and things are fine, but it made me wonder. Am I doing something inherently bad by using CCKD and Shadow files?
I was wondering how others on the list are set up.
Can you guys respond back with your thoughts on this?
Brian
Peter Glanzmann
2009-03-19 05:14:38 UTC
Permalink
Hi Brian

I'm running several Hercules instances on different machines, some of them
with CKD, some of them with CCKD DASD files. The only problem I once had,
was some corrupted CCKD files after a total system crash (the automatic
recovery for CCKD files - which starts the next time Hercules is powered on
- was not able to recover all of my DASDs).

While CKD files are very robust and have no problems after a system crash,
CCKD files seem to be more sensitive to system outages. But beneath that, I
never experienced any problems.

Peter

-----Ursprüngliche Nachricht-----
Von: hercules-390-***@public.gmane.org [mailto:hercules-390-***@public.gmane.org] Im
Auftrag von brian_westerman
Gesendet: Mittwoch, 18. März 2009 10:35
An: hercules-390-***@public.gmane.org
Betreff: [hercules-390] CCKD or not, Shadow Files or not

Hi,

I have been running for a few years now with my DASD in CCKD format and use
Shadow files. In speaking with another user last night, it came up that he
was not using CCKD because he felt it left the possibility to corrupt wide
open. He did not use Shadow files because he didn't really trust them
either.

I have never (until today) had any problems with CCKD (compressed DASD) and
shadow files. But today I decided to "merge" one of my DASD volumes shadow
file to the base. There was a "free space" issue, and while the sf- command
said everything was fine, when I next IPLed the system, most of that volume
was unusable. I was able to go back to my original BASE+SHADOW, and things
are fine, but it made me wonder. Am I doing something inherently bad by
using CCKD and Shadow files?

I was wondering how others on the list are set up.

Can you guys respond back with your thoughts on this?

Brian
Tony Harminc
2009-03-19 16:31:02 UTC
Permalink
I have been running for a few years now with my DASD in CCKD format and use Shadow files.  In speaking with another user last night, it came up that he was not using CCKD because he felt it left the possibility to corrupt wide open.  He did not use Shadow files because he didn't really trust them either.
I have never (until today) had any problems with CCKD (compressed DASD) and shadow files.  But today I decided to "merge" one of my DASD volumes shadow file to the base.  There was a "free space" issue, and while the sf- command said everything was fine, when I next IPLed the system, most of that volume was unusable.  I was able to go back to my original BASE+SHADOW, and things are fine, but it made me wonder.  Am I doing something inherently bad by using CCKD and Shadow files?
There is a thread starting on 2008-08-25 with the title "Trying to
understand CCKD integrity" that may shed some light on some of this. I
had run into a problem with CCKD data after a failure, and discovered
by reading code, and discussion in the thread that CCKD is much more
robust under many circumstances than one might expect. Contrary to my
initial impressions, it was designed with a great deal of thought to
integrity and recoverability. There remain some issues, but overall it
deals remarkably well with even quite ugly failures at the underlying
file system level.

Tony H.
Steve And Grace Bovy
2009-03-20 02:17:36 UTC
Permalink
if what you are saying is true, why have people been having problems ?????

_____

From: hercules-390-***@public.gmane.org [mailto:hercules-390-***@public.gmane.org] On
Behalf Of Tony Harminc
Sent: Thursday, March 19, 2009 9:31 AM
To: hercules-390-***@public.gmane.org
Subject: Re: [hercules-390] CCKD or not, Shadow Files or not
Post by brian_westerman
I have been running for a few years now with my DASD in CCKD format and
use Shadow files. In speaking with another user last night, it came up that
he was not using CCKD because he felt it left the possibility to corrupt
wide open. He did not use Shadow files because he didn't really trust them
either.
Post by brian_westerman
I have never (until today) had any problems with CCKD (compressed DASD)
and shadow files. But today I decided to "merge" one of my DASD volumes
shadow file to the base. There was a "free space" issue, and while the sf-
command said everything was fine, when I next IPLed the system, most of that
volume was unusable. I was able to go back to my original BASE+SHADOW, and
things are fine, but it made me wonder. Am I doing something inherently bad
by using CCKD and Shadow files?

There is a thread starting on 2008-08-25 with the title "Trying to
understand CCKD integrity" that may shed some light on some of this. I
had run into a problem with CCKD data after a failure, and discovered
by reading code, and discussion in the thread that CCKD is much more
robust under many circumstances than one might expect. Contrary to my
initial impressions, it was designed with a great deal of thought to
integrity and recoverability. There remain some issues, but overall it
deals remarkably well with even quite ugly failures at the underlying
file system level.

Tony H.





[Non-text portions of this message have been removed]
Mike Schwab
2009-03-20 04:33:56 UTC
Permalink
There was some bugs that were straightened out about 1.5 years ago.
Post by Steve And Grace Bovy
if what you are saying is true, why have people been having problems ?????
--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?
Tony Harminc
2009-03-20 15:03:47 UTC
Permalink
Post by Steve And Grace Bovy
if what you are saying is true, why have people been having problems ?????
That's an odd question; I didn't say or even suggest that there are no
problems with CCKD. I did try to point out that, given the underlying
infrastructure, the design and code do a remarkably good job of
recovering from bad situations, even in the face of unknown behaviour
on the part of the file system and even hardware that the CCKD volume
is hosted on. The particular bug I encountered (since fixed) was
unique to recovery handling on 64-bit OSs.

There were a couple of bugs reported with shadow file merge
processing, though I don't know their current status. And it remains
true that the recovery options are not as fully documented as might be
nice.

But really, as with any other data, if it's important to you, back it
up frequently and test your restore procedures. And think ahead of
time about likely failure modes; just as a wild example, if you have a
well backed up base file, and a well backed up top level shadow file,
but a lost intermediate shadow file, then you are pretty much screwed.

Tony H.
Ian S. Worthington
2009-03-21 17:40:39 UTC
Permalink
That's a good idea: One reason I use shadow files is that I can take a backup
of my changes every time I take down a system and then easily roll back to any
previous version just with an erase, unzip and copy.

Eliminating the compress overhead would be nice (I suspect).

Do I have to do the copy on the (base + shadow(s)) set, or can I choose to
have a compressed base and non-compressed shadows?

i

------ Original Message ------
Received: Sat, 21 Mar 2009 07:49:43 AM COT
From: "halfmeg" <opplr-***@public.gmane.org>
To: hercules-390-***@public.gmane.org
Subject: [hercules-390] Re: CCKD or not, Shadow Files or not
Post by halfmeg
<snip>
It appears (from my point of view), that if you don't mind the extra
space being used by non-compressed dasd and if you don't mind
backing up a LOT of space with each backup, ...
<snip>
but I'm sure people would do it even if only to make backing up the
volumes quicker).
<snip>
One way to reduce the backup of empty space and make backing up quicker
would be to use CCKD with no compression. DASDCOPY has the -0 option to allow
conversion between various types of dasd to CCKD no compression. One could
Post by halfmeg
dasdcopy -0 -o CCKD mvsres.xxx mvsres.xxx.CCKD
in order to see how much space would be saved for backups.
One of the MVSRES packs available on the web went from 326,861,312 down to
31,226,244 using the above.
Post by halfmeg
Phil
------------------------------------
http://groups.yahoo.com/group/hercules-390
http://www.hercules-390.org
Yahoo! Groups Links
Continue reading on narkive:
Loading...