'Scaled' would be the appropriate word. The point is that what the raw numbers are is of zero significance. It's an arbitrary code that represents something - and it's what it represents that matters. You could translate them all to ASCII strings. It would make them 'bigger' but wouldn't change what they represent. The numbers represent per-pixel charge, which in turn measures per-pixel exposure. The various settings in the signal chain affect the precision and noise in that measurement, but they don't change what is the per-pixel exposure. If you do change them, then the rendering engine needs to scale them differently, but again this is arbitrary, it doesn't stop it haveing to do all the processes requited to turn that set of per-pixel exposure measurements into a viewable image.
It's not really analogous, for (at least) two reasons. Firstly, your audio system's volume control occurs after the coded audio signal has been decoded into a representation suitable for listening, not before. Secondly, it's a completely different situation. In an audio system the input and output represent the same type of quantity - instantaneous sound pressure. In photographic imaging the input and output are completely different types. The input is measurements of localised light energy, the output is locations in a colour space, which in turn is a description of how a human being should see the image. Turning the one into another requires much more processing than 'gain', and ascribing 'gain' as being solely responsible for one component of the colour space is misleading. Variable gain or no, the output value of all of the components need to be computed.
I don't think it's pedantry. When people are striving to understand something the make inferences from the words that are used to explain them. If you say 'ISO is gain' the general inference seems to be that ISO 'gains' light in some manner. This of course is completely wrong and leads to further misunderstandings, thus the word is to be avoided, especially as the statement that ISO is gain is factually incorrect.