As mentioned, the battery is far and large the biggest expense. The boost/charge circuits, ready to solder to a battery pack, are around $2 each. Why not use two of them - one to charge one to boost and power the device, with the batteries in between. This would allow for charging and discharging at the same time. Might introduce some extra noise to the output, but the battery should filter it out. Worst case, might need to add a capacitor to the output circuit.