I think it depends on your goals. For a fun DotSam and/or DotStr playthrough, agility can take point and you build down from there. I found out the hard way en route to WotN in DotD and Wise that your agility rating (was also initially a point of pride for me) just wasn't getting it done -- couldn't even survive a margin of error to allow for chip or stun, it felt. I also should say for someone like you who's bested Bb at SL1 *tipping of hunter hat, you may well be able to pull through; to that breed of player, I saw dudes in Dragon Ninja (lightweight set) at the highest levels so it's more a cautionary tale for most of how out of whack AI HP and DMG scales as you progress. It's actually a bit cruel. Lol.
This is why I recommended to Coyote always to side with Ki DMG (cores, spirits, claws, tempering et cetera) over raw because the former scales (much) more manageably in the long game. Stam regen is something I had to relearn in DotD with combo-counting, for example, to ensure I avoid winded status penalties. You really need be able to eat some Mike Tyson-grade shots later.