Cell placement bounding box

The bounding box getter of the instance oaInst::getBBox() returns the bounding box around the FULL contents of the instance (including any physical pins and P/G pins on the top and bottom sides of the instance)

The SIZE of an instance as it is defined in a LEF can be obtained using the oaSnapBoundary object which can be obtained from the master of the instance.

oaDesign* master = inst->getMaster();
oaSnapBoundary* boundary = oaSnapBoundary::find(master->getTopBlock());
oaBox box;   // <=== four oaPoint objects that describe the bounding box.
boundary->getBBox(box);

The coordinate system used in oaSnapBoundary is the block coordinate system, not its parent design.

oaSnapBoundary in documentation,

An oaSnapBoundary must be a rectangle. Placement tools use the oaSnapBoundary to place instances of the design in rows, abutting the oaSnapBoundary edges and aligning the oaSnapBoundary corners. The SnapBoundary will provide an unambiguous lower-left corner that will control the exact placement location.