Its not recommended to use fieldset any more, Fieldset and legand are very hard to style and its not recommeded to use with Drupal 8 any more. Instead its replaced with more cross browser way, which is the HTML5 Details & Summary tags...

Here is how to use <details> instead of fieldset through FAPI

Drupal 7

php
 
 $form['vertical_tab_settings'] = array(
   '#type' => 'vertical_tabs',
   '#default_tab' => 'edit-slideshow',
  ) ;
  $form['colors'] = array(
   '#type' => 'fieldset',
   '#title' => t('Colors'),
   '#collapsible' => TRUE,
   '#collapsed' => TRUE,
   '#group' => 'vertical_tab_settings',
  );
  $form['colors']['title_color'] = array(
   '#type' => 'textfield',
   '#title' => t('Title color'),
   '#field_prefix' => '#',
  );

Drupal 8

php
 
 $form['vertical_tab_settings'] = array(
   '#type' => 'vertical_tabs',
   '#default_tab' => 'edit-slideshow',
  ) ;
  $form['colors'] = array(
   '#type' => 'details',
   '#title' => t('Colors'),
   '#collapsible' => TRUE,
   '#collapsed' => TRUE,
   '#group' => 'vertical_tab_settings',
  );
  $form['colors']['title_color'] = array(
   '#type' => 'textfield',
   '#title' => t('Title color'),
   '#field_prefix' => '#',
  );

One more Drupal 8 example

php
$form['details2'] = array(
    '#type' => 'details',
    '#title' => t('Details2'),
    '#collapsible' => TRUE,
    '#collapsed' => TRUE,
    '#states' => array(
      'expanded' => array(
        ':input[name="expanddetails2"]' => array('value' => 'expand'),
      ),
    ),
  );
  $form['details2']['item'] = array(
    '#type' => 'item',
    '#title' => t('Do you seee me?'),
    '#markup' => t('If you can see me the details is not collapsed!'),
  );
  $form['expanddetails2'] = array(
    '#type' => 'textfield',
    '#title' => t('Change details state'),
    '#description' => t("Type 'expand' to expand Details2."),
  );

Thats it :) good luck !