For banks, credit derivatives are back in the spotlight—and flashing warning signals.

The market for credit-default swaps has shrunk to a fraction of the size it was before the financial crisis. But these securities, created to insure against default, still can send shivers through markets as a closely watched measure of banks' creditworthiness.