Most of what I know about that comes from Brian Bagnall's books (e.g. Commodore: A company on the edge). It's covered in the first chapter. I don't know of any preserved documents - if they exist they'd belong to Motorola, and I'd expect the leavers would have preferred to ditch any such documents (though part of the reason for the settlement with Motorola was that one of the engineers stupidly taking 6800 documentation with him). This [1] summarizes it. Here [2] is an interview by Computer History Museum of Chuck Peddle. It matches up with the first chapters in Bagnalls book very closely. From about 50 minutes in he talks about his meetings w/Motorola customers that told him the 6800 was too expensive. From about 58m in they talk about basically setting a size metric for the chip to meet a low enough price point. So the starting point was what they could fit in small enough space. You might also like the Computer History Museum interview w/Bill Mensch [3] [1] [link] [2] [link] [3] ▶ this video
in “6502 “Illegal” Opcodes Demystified” · View on HN